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I.   Introduction 

In  recent  years  schools  of  management  and  industrial  managers  have  focused 
their  attention  on  the  development  of  quantitative  approaches  to  management 
problems.   Inventory  control,  production  scheduling,  capital  budgeting,  cost- 
effectiveness  analysis  and  other  areas  have  been  the  principal  beneficiaries 
of  this  effort.   Although  we  have  numerous  reports  of  successful  implementa- 
tiorv  it  is  generally  true  that  the  theory  has  outstripped  our  capability  to 
implement  these  techniques. 

This  report  examines  the  implementation  of  cost-effectiveness  analysis 
by  the  Army  Materiel  Command.   Specific  attention  has  been  focused  on  the 
interface  between  the  quantitative  methodology  and  the  user  of  the  analysis 
results,  the  decision  maker.   Successful  implementation  depends  upon  a 
successful  coupling  at  this  interface.   The  coupling  mechanism  between  the 
decision  maker  and  the  analyst  who  applied  the  methodology  is  the  information 
flow,  formal  as  well  as  informal,  between  them.   Consequently,  much  of  the 
discussion  to  follow  will  deal  with  methods  for  bringing  the  needs  of  the 
decision  maker  to  bear  on  the  objectives  of  the  analyst,  with  the  methodology 
itself,  and  with  the  role  of  the  information  system  requirements  in  such 
context 
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II.   Obiectlves 

The  title  of  this  research  is  "The  Sensitivity  of  Management  Decisions 
to  Quantitative  Models  and  Methods  for  Improvement  in  Such  Models  through 
the  Management  Information  and  Control  System".   This  title  reflects  the 
general  objectives  chosen  for  the  project.   The  intent  was  to  satisfy  both 
the  interests  of  the  Army  Materiel  Command  and  the  Management  Information  and 
Control  groups  at  MIT.   In  approaching  our  research  we  delineated  eight 
subsidiary  objectives.   These  were: 

1.  To  examine  the  sensitivity  of  decisions  to  quantitative 
analysis.   That  is,  to  examine  the  effect  of  cost- 
effectiveness  studies  on  management  decisions,  and  to 
understand  the  role  of  studies  in  the  larger  decision 
process . 

Specif icallyj we  were  interested  in  whether  or  not  cost- 
effectiveness  studies  ha  ve  a  major  influence  on  the 
decision  makers  in  the  execution  of  their  responsibilities. 
Studies  may  merely  confirm  or  substantiate  the  decision 
maker's  initial  judgement,  or  be  the  sole  determinant 
of  the  decision,  or  may  combine  the  decision  maker's  value 
judgements  and  the  many  objective  factors  into  the  deci- 
sion analysis. 

2.  To  examine  the  role  of  the  decision  maker  in  a  decision 
process  which  is  based  on  quantitative  analysis  and  to 
determine  in  what  way  his  time  can  most  effectively  be  ap- 
plied to  the  systems  analysis.   There  are  numerous  con- 
tributions which  a  decision  maker  can  make  at  the  beginning 
of  the  study,  during  the  execution  of  the  study,  and  in  the 
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interpretation  of  the  results.   We  were  interested  in  the 
effectiveness  of  the  decision  maker's  participation  in 
each  of  these  areas. 

3.  To  recommend  ways  for  improving  the  consistency  between  the 
objectives  of  the  decision  maker  and  those  pursued  in  the 
study.   In  many  cases,  quantitative  analyses  tend  to  answer 
questions  which  are  either  framed  in  different  terminology 
than  that  of  the  decision  maker  or,  in  fact,  answer  dif- 
ferent questions  than  the  decision  maker  had  in  mind. 

4.  To  recommend  ways  for  improving  the  timeliness  of  the  study 
results  by  reducing  the  elaps6d  time  between  the  request 
for  information  and  the  completion  of  the  study.   Success 

in  this  area  would  also  reduce  the  disruption  of  the  analytic 
process  to  reasonable  bounds. 

5.  To  suggest  methodological  approaches  to  the  systems  analysis 
in  order  to  provide  more  flexibility  and  enhance  its  utility 
to  the  decision  maker   Methodological  approaches  can 
become  over  quantitative  and  rigid  attributes  which  may  con- 
strain their  application  and  usefulness. 

6.  To  recommend  ways  for  increasing  the  decision  maker's  contri- 
bution to  the  interpretation  of  the  results  of  systems 
analysis  studies.   That  is,  suggest  methods  by  means  of 
which  the  decision  maker  can  impart  his  judgements  and  values, 
as  inputs,  to  the  SA  findings.   Many  times,  in  a  study,  sig- 
nificant value  judgements  are  made  that  do  not  necessarily 
represent  the  values  of  the  decision  maker.   It  was  our 
objective  to  find  ways  of  insuring  that  the  value  structure 
of  the  decision  maker  was  strongly  represented  in  the  inter- 
pretive process. 


7.  To  improve  the  overall  efficiency  of  the  study  process,  both 
in  terms  of  goals  and  methodology,  through  the  management 
information  system.   To  propose  new  information  systems  to 
complement  the  present  managerial  system  and  provide  feed- 
back for  efficient  and  effective  management.   From  the 
identification  of  a  problem,  to  the  recommendation  of  a 
solution,  to  the  action  taken  with  respect  to  that  recom- 
mendation, there  are  many  information  sources  and  channels 
which  must  be  considered  for  the  most  effective  decision 
making  process.   Consequently,  it  was  our  intention  to  find 
ways  of  improving  the  efficiency  of  this  communication 
process. 

8.  To  suggest  ways  for  increasing  the  transferability  of  benefits 
from  studies.   Beyond  the  decision  oriented  recommendations  of 
a  study,  there  is  a  large  amount  of  significant,  widely  ap- 
plicable information,  such  as: 

New  Data 

New  Methodology 

Subsidiary  weapon  concepts 

Criteria  for  measuring  decision  quality  and 
future  performance 

A  study  built  on  sound  methodology,  updated  to  reflect  the 
current  state  of  knowledge  (in  terms  of  data,  analytical 
methods,  and  environmental  assumptions)  and  properly  clas- 
sified for  easy  access,  can  serve  as  an  invaluable  source 
document  for  future  needs.   We  wanted  to  examine  ways  of 
transferring  this  knowledge  to  other  relevant  situations. 


III.   Assumptions 

To  put  these  objectives  in  the  proper  framework  and  to  describe  the 
basis  for  our  study  methodology,  it  is  necessary  that  we  delineate  our 
study  assumptions.   We  assume  that: 

1.  Top  managerial  and  decision-making  talents  exist.   The  major 
aspect  that  limits  the  scope  of  these  talents  is  the  inabil- 
ity of  the  decision  makers:   (a)  to  cope  with  the  large 
amount  of  unstructured  data,  i.e.  this  inability  is  mainly  due 
to  the  lack  of  time  and,  to  a  lesser  extent,  to  the  lack  of 
methodological  expertise  to  digest,  organize,  filter, 
condense,  evaluate  and  extract  information  out  of  the  mass 
of  relevant  and  irrelevant  environmental  data  (b)  to  readily 
change  the  structure  of  the  decision  system,  as  dictated  by 
environmental  changes. 

2.  In  order  to  be  able  to  cope  with  both  data  and  adaptively 
changing  environments,  the  manager  must  know  enough  about 
relevant  theories,  methodologies  and  systems  analysis,  and 
about  the  tools  used,  to  appreciate  the  power  and  limita- 
tions of  the  technology  and  become  the  interface  between  the 
tools,  the  methodology  and  the  decision  situations.   This, 
however,  does  not  imply  that  the  decision  maker  must  be  an 
expert  technician  or  system  analyst  himself. 

3.  The  decision  maker  should  be  able  to  derive  the  necessary 
information  needed  for  the  decisions  he  faces,  from  the 
following: 
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a)  the  study  objectives 

b)  the  assumptions  which  impinge  on  the  data  and  the 
methodology  used 

c)  the  description  of  the  methodology  used 

d)  the  description  of  the  critical  parameters  considered 
and  how  they  constrain  the  results 

e)  the  results  and  conclusions 

4.  The  decision  maker  is  a  planner,  not  a  fire-fighter.   There- 
fore, the  greatest  payoff  for  his  time  investment  will  come, 
if  it  is  spent  on  the  definition  and  planning  phases  of  the 
study  process  and  on  the  provision  of  control  to  the  indi- 
vidual studies.   In  this  way  he  will  not  have  to  monitor 
continuously  the  critical  environmental  assumptions  which 
affect  the  objectives  of  the  studies  and  the  study  process 
itself.   The  information  system  will  provide  him  with 
signals,  if  the  critical  assumptions  on  which  he  bases  his 
plans  are  violated,  indicating  a  necessity  for  replanning. 

5.  Through  a  management  information  system,  signals  can  be 
generated  to  complement  the  decision  maker's  personal 
supervision  and  also  provide  preventive  causal  diagnosis. 

6.  The  decision  maker  has  sufficient  authority  to  implement 
changes  in  the  present  system  and  the  present  study 
process 

7.  Learning  from  successive  cost-effectiveness  models  is  pos- 
sible and  the  transfer  of  these  benefits  is  desirable. 


IV.   Theoretical  Framework 

Planning  and  control  theory  suggests  that  the  formalization  of  decision 
models  is  necessary  for  learning  and  sequential  adaptation  to  changes  in  the 
environment. 

In  particular,  with  respect  to  planning, there  is  a  lot  to  be  learned 
from  experience  if,  somehow,  we  could  capture,  document,  evaluate,  and 
preserve  it  for  later  use.   The  fact  that  each  particular  planning  op- 
portunity appears  to  be  unique  has  prevented  the  appreciation  of  another 
fact,  that  planning  processes  have  many  elements  in  common  irrespective  of 
the  particulars  of  the  setting  to  which  these  are  applied. 

Unfortunately,  within  most  organizations,  planning  is  carried  out  as  an 
ad  hoc  activity  and  completely  divorced  from  the  on-going  information  and 
control  system.   This  practice  encourages  undue  reliance  on  intuition,  and 
memory,  and  at  the  same  time  ignores  the  presence  of  a  framework  which  can 
aid  learning.   The  latter  can  occur  through  the  discovery  of  new  patterns 
of  relationships,  and  in  the  process  of  review  of  the  various  assumptions, 
and  the  cause-effect  relationships  underlying  the  various  elements  in  the 
planning  process.   A  formal  linkage  between  an  explicitly  stated  plan  and 
the  on-going  information  and  control  system  will  ensure  that  the  data,  which 
are  necessary  to  validate  and  implement  the  chosen  plan,  are  collected. 
At  the  same  time, this  linkage  will  allow  a  review  of  the  whole  process  so 
that  the  organization  may  learn  from  experience. 
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It  is  for  the  above  reasons  that  we  consider  formal  planning  and  deci- 
sion making  a  necessity.   As  used  here,  formalization  includes  at  least 
three  elements: 

(a)  The  explicit  statement  for  any  plan  of:   the  critical  under- 
lying environmental  assumptions;  the  dominant  cause-effect 
relationships;  the  consequences  of  such  on  the  critical 
decision  which  have  to  be  made  and  on  the  operations  which 
are  necessary  to  carry  out  the  plan;  the  definition  of  the 
necessary  signals  or  indicators  which  will  provide  managers 
with  the  timely  feedback  for  replanning;  and  the  details  of 
the  measurements  which  will  generate  the  indicators.   (always 
bearing  in  mind  the  sensitivity  of  management  plans  and 
decisions  to  the  refinement  in  these  measurements) 

(b)  The  definition  of  the  information  and  control  system  which 
will  apply  the  measurements  and  generate  the  indicators 
which  will  facilitate  learning,  planning,  and  control  on 
an  exception  basis. 

(c)  The  mechanism  for  a  formal  review  and  replanning  procedure 

at  regular  intervals.   This  procedure  will  guarantee  that: 

(i)  A  plan  for  planning  exists,  and  that  this 
plan  is  not  solely  dependent  on  the 
signals  generated  by  the  information  and 
control  system. 

(ii)  The  search  for  planning  alternatives  is  not 
limited  to  variations  of  existing  plans. 

To  summarize  then,  formalized  planning  in  our  terms,  imposes  a  discipline 
for  explicit  recognition  of  alternatives,  assumptions,  cause-effect 
relationships  and  the  consequences  of  such,  on  managerial  decisions.   It 
also  forces  on  the  manager  some  rationality  and  consistency  in  dealing 


with  alternatives  and  going  through  the  various  steps  in  the  planning  process. 
It  provides  for  the  definition  and  generation  of  information  to  facilitate 
replanning,  and  it  ensures  the  manager  an  opportunity  for  reviewing  the  total 
process.   All  these  attributes  of  formalized  planning  are  very  conducive  to 
learning  from  experience.   Finally,  the  link  between  plans  and  the  on-going 
information  system  provides  organizations  with  a  unique  opportunity  to 
store  planning  intelligence  and  to  facilitate  the  transferability  of  planning 
knowledge . 

In  the  context  of  this  project,  formal  models  are  necessary  because: 

1.  The  decision  maker  is  faced  with  large  quantities  of  raw 
data  in  unstructured  form  which  must  be  analyzed,  filtered 
and  condensed.   Without  a  framework,  it  will  be  close  to 
impossible  to  create  a  semblance  of  order  out  of  chaos. 

2.  The  decision  maker  needs  to  have  access  to  adaptive  models 
of  the  world  against  which  he  can  test  alternative  plans. 

The  explicit  recognition  of  environmental  assumptions  and  the 
underlying  cause-effect  relationships  will  facilitate  the 
testing  and  evaluation  of  alternatives. 

3.  The  system  must  provide  timely  response  to  the  changes  which 
occur  in  the  planner's  model  of  the  world.   This  can  be 
accomplished  if  critical  assumptions  are  monitored  and 
deviations  from  acceptable  nomis  are  reported  to  the 
manager. 

4.  Specific  statements  of  cause  and  effect  relationships  are  neces- 
sary for  preventive  diagnosis.   A  formal  system,  not  only  ex- 
plicitly states  such  critical  cause-effect  relationships,  but 
also  provides  a  link  between  plans  and  the  management  informa- 
tion system.   The  signals  generated  by  the  control  system  can 

be  further  amplified  to  include  consequences  of  projected 
deviation  from  expected  norms. 
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5.   Transferability  is  achieved  if  there  is  standardization  of 
data  format,  classification  of  models,  hardware,  software 
and  management  information  systems.   It  has  been  already 
stated  that  the  formalization  of  plans,  and  the  linking  of 
such  to  on-going  information  and  control  systems,  provides 
among  other  advantages  a  unique  opportunity  for  pattern 
recognition  and  the  development  of  organization-wide 
intelligence.   Some  other  consequences  of  such  a  successful 
linkage  will  be  the  determination  of  the  necessary  input  data 
for  testing   and  implementation  of  the  models,  and  also  the 
standardization  of  terminology,  of  classifications,  and  of 
querry  languages. 

We  must  admit  that  the  road  leading  to  the  formalization  of  planning 

models  and  the  linking  of  these  to  the  managerial  information  and  control 

system  are  not  completely  free  of  problems.   One  must  be  prepared  to 

overcome: 

(a)  The  aversion  of  people  to  state  explicitly  their  assumptions 
and  thought  processes,  because  of  fear  that  later  history 
may  prove  them  wrong. 

(b)  The  possible  tendency  of  model  builders  to  use  complicated 
methodological  approaches  which  are  either,  unnecessarily 
complex,  or  little  understood  by  the  users  of  these  models. 

(c)  The  difficulty  of  validating  models  both  with  respect  to  their 
structure  and  the  relevance  of  the  parameters. 

(d)  The  problems  inherent  in  updating  models  and  their  re-use. 

(e)  The  difficulties  involved  in  developing  relevant  input  data 
to  a  control  system  of  formal  plans,  and  resolving  all  the 
issues  associated  with  the  timing  of  replanning  activities. 


-li- 
on the  basis  of  our  research  under  this  project  and  the  elucidation  that 
it  brings  to  some  of  our  prior  experiences,  we  are  motivated  to  say  that,  the 
major  problem  with  formal  models,  is  not  so  much  with  the  models  themselves 
(methodology)  but  in  the  difficulty  of  providing  an  adequate  decision  maker- 
model  interface.   There  are, of  cours^  many  facets  to  these  interfaces.   Any 
formal  system  must  involve  the  decision  maker,  the  model  itself,  the  analyst, 
and  the  data  base  required  by  the  objectives  of  the  planning  process.   If 
the  interfaces  between  each  of  these  components  are  not  designed  properly, 
the  decision  process  will  no  doubt  suffer.   In  addition,  in  many  real  situa- 
tions the  model  has  to  be  constructed  and  this  may  involve  technicians  of 
various  specialities.   In  a  real   dynamic   environment  then,  in  addition  to 
the  interface  problem  between  such  systems  analysts  and  the  rest  of  the 
components  mentioned  above,  there  is  a  further  problem  of  successfully 
resolving  the  interfaces  between  model  builders  themselves . 

Because  of  the  above  reasons,  and  in  the  context  of  most  AMC  needs: 

1.  Models  must  allow  the  decision  maker  to  readily  test  var- 
ious values  of  parameters.   They  must  also  permit  and 
facilitate  choice  of  the  level  of  output. 

2.  Models  must  specify  the  parameters  and  the  various  threshold 
constraints  that  are  critical  for  decisions,  so  that  the 
decision  maker  can  introduce  his  value  system  by  choosing 
among  these  factors. 

3.  Models  must  monitor  the  critical  relations  between  the  en- 
vironment, the  objectives,  and  the  task,  and  also  incorporate 
in  them  the  ranges  over  which  critical  values  of  these  can 
vary  without  affecting  the  decision  situation. 
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4.   The  critical  cause-effect  relationships, some  of  which  are 
implied  under  Point  3  above,  must  be  explicitly  stated  to 
capture  as  many  of  the  dynamic  aspects  of  the  modeled  sit- 
uation as  possible.   In  this  way,  the  decision  maker  would 
be  able  to  use  the  model  as  an  active  tool  in  exploring  his 
problem  and  the  alternative  solutions  to  it. 


V.   Criteria  of  Study  Effectiveness 

One  of  the  first  needs  recognized  by  the  study  group  was  to  develop  some 
criteria  by  which  the  various  studies,  made  available  to  us  could  be  evaluated, 
We  were  unable  to  do  this  except  on  a  very  rudimentary  basis  at  the  begin- 
ning of  our  study   We  ultimately  developed,  however,  seven  criteria  for 
evaluating  the  quality  and  utility  of  the  cost-effectiveness  studies.   It 
should  be  emphasized  that  our  viewpoint  was  quality  and  utility  from  the 
standpoint  of  the  decision  maker.   Other  viewpoints  such  as  the  elegance 
of  the  solution  or  other  similar  criteria  had  little  bearing  in  this  study. 

The  first  criterion  is  the  time  required  for  the  decision  maker  to 
comprehend  the  essence  of  the  analysis.   If  a  decision  maker  is  to  make  a 
decision  based  on  the  analytical  efforts  of  others,  he  must  be  able  to 
comprehend  the  nature  of  the  analysis  which  they  have  made  and  to  do  this 
in  significantly  less  time  than  it  took  those  who  performed  the  analysis. 
Specifically,  the  decision  maker  must  not  be  required  to  read  and  evaluate 
all  aspects  of  each  study. 

The  second  criterion  is  the  relevance  of  the  study  to  the  decision  which 
must  be  made.   Needless  to  say,  a  study  which  answers  questions  unrelated  to 
the  decisions  which  must  be  made  has  little  value.   For  example,  if  a  deci- 
sion maker  must  make  a  procurement  decision,  a  study  which  shows  only  that 
the  particular  kind  of  truck  is  better  under  certain  circumstances  than  all 
other  kinds  of  trucks,  and  not  the  quantity  to  be  procured,  is  deficient. 
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The  third  criterion  is  the  amount  of  information  required  to  make  a 
decision  beyond  that  provided  by  the  study.   One  of  the  traps  into  which 
an  operations  analyst  can  fall  is  treating  only  those  parameters  which  are 
susceptible  to  quantitative  analysis,  leaving  the  qualitative  parameters 
to  the  decision  maker  and  others  for  input  and  evaluation.   Treating  these 
qualitative  parameters  after  the  systems  analysis  can  lead  to  a  follow-on 
study  as  large  as  the  original  one. 

The  fourth  criterion  is  the  timeliness  of  the  results  of  a  study    In 
most  cases  a  decision  must  be  made  at  a  certain  point  in  time  even  with  in- 
complete information.   If,  for  some  reason,  a  study  is  inadequate  and  revi- 
sions must  be  made,  the  making  of  the  decision  will  necessarily  be  delayed. 
If  this  delay  continues,  new  alternatives,  and  new  projections  of  the 
enemy's  threat  must  be  re-evaluated,  creating  obsolescence  in  the  original 
objectives  of  the  study. 

The  fifth  criterion  is  the  ease  with  which  the  study  can  be  revised. 
While  it  is  possible  for  a  decision  maker  to  specify  at  the  beginning  those 
questions  which  he  wishes  to  be  answered,  many  considerations  change  in  the 
process  of  performing  the  study.   New  problems  and  questions  arise  making 
some  revisions  in  the  study  desirable.   Ease  of  revision  might  be  measured 
in  terms  of  the  time  required  to  revise  the  study,  the  cost  required,  and 
the  useability  of  the  old  computer  programs  and  routines  developed  for  the 
original  study.   In  any  event,  in  conducting  and  evaluating  any  series 
of  studies,  it  should  be  recognized  that  not  only  in  the  short  term  when 
the  decision  maker  is  considering  the  study  for  the  first  time,  but  also 
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in  the  longer  term,  there  will  be  constant  needs  for  revision.   These  needs 
should  be  reflected  in  the  choice  of  study  methodology  and  the  mode  of 
presenting  the  results. 

The  sixth  criterion  is  the  ease  in  making  parametric  changes  and  asking 
questions  of  the  model.   Most  decision  makers  will  need  to  ask  certain 
questions  which  are  not  specifically  answered  in  the  presentation  of  the 
results.   Principally,  this  is  to  enable  them  to  understand  the  structure 
and  relationships  within  the  model  and  the  sensitivity  of  the  results  to 
a  change  in  the  assumptions. 

The  seventh  criterion  is  the  methdological  appropriateness.   It  is  pos- 
sible that  in  some  studies  methodologies  too  complex  or  too  simple  for  the 
problem  will  be  used.   In  economies  we  use  a  criterion  of  simplicity  to 
choose  between  models.   Perhaps  a  similar  criterion  is  also  valid  in  the 
systems  analysis  area. 


VI.   Study  Procedure 

Analysis  of  the  Problem 

The  study  was  initiated  with  a  series  of  round  table  discussions  in  which 
various  objectives  and  study  approaches  were  considered.   Relevant  sources 
of  literature  were  also  sought  at  this  time.   After  a  reasonable  period  of 
time  during  which  the  study  team  was  built  and  some  tentative  ideas  developed, 
discussions  were  held  with  General  Bunker. 

After  these  discussions,  it  was  decided  to  examine  the  cost-effectiveness 
process  from  the  viewpoint  of  its  information  systems.   Specifically,  we  chose 
to  look  at  the  interfaces  between  the  analysts,  the  study  itself,  the 
experience  from  previous  studies,  and  the  decision  maker.   We  also  wanted  to 
examine  the  impact  of  the  methodology  on  the  information  process. 

Pilot  Study 

In  order  to  gain  a  broad  perspective  of  the  study  process,  it  was  decided 
to  examine  a  large  sample  of  studies.   To  sharpen  our  approach,  a  pilot  study 
of  the  "Effectiveness  Comparison  of  4.2  Mortar  Ammunition  M329A1  vs.  XM571" 
and  the  "Cost  Performance  Analysis  of  M656,  M520,  M542A  and  M548  Vehicles" 
was  undertaken.   From  the  information  learned  in  these  studies^ a  series  of 
hypotheses  (see  attachment  1)  were  developed  for  the  larger  analysis, and  a 
questionnaire  was  developed  to  test  these  hypotheses  (see  attachment  2). 
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FuU  Scale  Study 

Using  the  following  general  criteria,  and  with  the  aid  of  Tom  Shirata,  we 

selected  the  studies  which  appear  in  attachment  3  for  analysis. 

Completed  rather  than  in  process  studies 

Completed  sufficiently  recently  that  personnel  can  still 
be  interviewed 

Weapon  procurement  studies  rather  than  strategy  studies 

Include  various  types  of  study  groups-- industry,  project 
manager,  experienced  in-house  study  groups,  etc. 

Of  different  cost  magnitudes  and  depths  of  analysis. 

The  studies  were  sent  to  MIT  where  they  were  read.   Interviews  were  then 
set  up  with  the  study  manager  and  where  possible,  the  cognizant  review  agency 
and  the  decision  maker.   Interviews  lasted  from  45  minutes  to  an  hour  and  a 
half.   The  questions  in  the  questionnaire  were  first  asked  and  then  some 
time  was  given  to  free  discussion. 

Analysis 

The  first  analysis  approach  which  we  undertook  was  descriptive.   A  deci- 
sion model,  a  document  flow  model,  and  an  organizational  flow  model  of  the 
process,  as  we  observed  them,  were  developed.   Each  of  these  was  analyzed  and 
then  an  improved  decision  model  was  developed. 

Our  second  analysis  approach  was  the  quantitative  analysis  of  the  question- 
naire data  in  which  two  basic  steps  were  taken: 

Studies  were  ranked  on  the  basis  of  quality 

This  measure  of  quality  was  used  as  a  basis  for  testing 
our  hypotheses  about  study  effectiveness. 
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The  quality  ranking  was  developed  by  employing  the  best  operational 
measures  of  the  criteria  of  effectiveness    which  we  had  available. 
These  measures  were: 

(1)  Was  the  study  re-done? 

(2)  Was  the  recommendation  of  the  study  followed  by  the 
decision  maker? 

(3)  Was  the  structure  and  presentation  of  results  of  the  study 
rated  high  or  low?   (In  essence,  we  evaluated  the  content 
of  each  study  to  determine  whether  or  not  all  the  elements 
which  a  decision  maker  might  search  for  in  a  study  were 
easily  identifiable.   The  studies  were  then  split  into  two 

equal  categories--elements  were  easy  to  find,  elements 

(2) 
were  difficult  to  find.) 


Each  study  was  given  a  point  score  of  0  or  2  for  the  first  two  criteria 
and  of  0  or  1  for  the  third  criteria.   These  ratings  were  augmented  by  over- 
all study  evaluations  for  nine  studies  which  were  obtained  from  AMC  personnel. 
If  the  study  was  considered  good,  it  received  2  points.   If  it  was  poor,  it 
received  0  points.   The  overall  subjective  evaluations  agreed  well  with  the 
three  objective  measures.   Where  data  was  missing, a  0  was  given.   The 
scores  from  each  of  these  criteria  evaluations  were  then  summed  to  form  an 
overall  ranking  index.   Based  on  this  index, the  studies  were  then  split 
into  four  categories,  excellent ,  good,  fair,  adequate.   All  the  studies  in 
each  category  were  about  equal.   (see  attachment  4  for  the  ranking).   There 


See  section  on  criteria  of  effectiveness. 

2 
See  Findings  of  the  Interface  section  for  a  more  thorough  discussion 


of  this  point. 
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are  a  number  of  deficiencies  with  this  ranking  of  studies,  both  in  the  measures 
which  were  used  and  in  the  data  which  in  some  cases  was  incomplete.   However, 
this  did  provide  some  measure  for  evaluating  our  hypotheses. 

We  next  ran  Spearman  rank  order  correlations  between  our  hypothesized 
causal  parameters  and  the  just  described  effectiveness  measures  to  test  our 
hypotheses . 

The  third  analysis  approach  was  interpretation.   We  had  collected  a  great 
deal  of  notes  and  written  material  which  favorably  supported   and  suggested 
improvements  for  the  system.   We  tried  to  take  this  material  and  integrate 
it  with  the  two  above  analyses  as  well  as  with  our  own  theoretical  knowledge 
and  draw  out  those  other  findings  which  seemed  important. 

Finally,  we  examined  the  study  methodologies  through  reading  the  studies, 
drawing  from  our  interviews,  and  examining  how  this  methodological  structure 
might  be  made  more  compatible  with  the  decision  maker's  information  needs. 


1       I 


VII.   Findings  and  Recommendations 

(A)  Interface  Analysis 

Introduction 

Cost-effectiveness  analysis  is  not  entirely  a  new  innovation  in  the  evalu- 
ation of  trade-off  decisions.   Significant  applications  of  this  methodology 
were  made  during  the  Second  World  War,  and,  with  the  passage  of  time, 
managers  have  found  the  technique  more  and  more  useful.   As  the  applica- 
tion has  broadened,  managers  have  formed  task  groups  whose  specific  func- 
tions are  to  undertake  these  more  sophisticated  and  complex  analyses.   In 
the  early  days  the  decision  maker  typically  kept  his  staff  very  close  to 
him  and  made  significant  contributions  to  the  analysis  itself.   As  time 
has  passed,  however,  the  necessity  for  these  studies  has  grown  considerably. 
To  accommodate  this  demand,  autonomous  organizations  have  been  created 
specifically  to  undertake  these  studies.   With  the  advent  of  this  speciali- 
zation  and  autonomy  from  the  decision  maker,  the  study  process  has  changed. 
New  communication  channels  and  technical  interfaces  have  been  created  and 
analysts  have  begun  to  assume  power  that  previously  belonged  only  to  the 
decision  makers.   The  decision  makers  role  has  tended  to  become  more  one 
of  initiating  studies  and  acting  on  their  recommendations.   Meanwhile,  the 
analyst  has  become  a  distant  interpreter  of  the  decision  maker's  needs  and 
judgements,  and  a  practitioner  of  the  new  complex  decision  technology. 

This  new  structuring  of  the  process  seems  to  have  resulted  in  the  studies 
becoming  less  responsive  to  the  manager^  decision  requirements,  both  in  terms 
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of  timeliness  and  adequacy.   In  this  particular  section,  therefore,  we  have 
chosen  to  directly  examine  the  interface  problems  in  the  present  day  study 
process.   As  can  be  seen  from  the  figure  below, 

Interface  Model 


I  Analyst 


Study  and 
Data  Bank 


there  are  five  very  important  Interfaces  which  must  be  considered  in  the  cur- 
rent study  process. 

The  first  (1)  is  the  interface  between  the  decision  maker  and  the  analyst. 
Here  the  concern  is  with  the  transmission  of  the  decision  requirements,  the 
study  objectives,  and  the  relevant  value  judgements  from  the  decision  maker 
to  the  analyst.   Too  often  perhaps  a  directive  of  several  pages  is  thought 
to  be  sufficient. 

The  second  (2)  is  the  interface  between  the  decision  maker  and  the  study. 
If  a  decision  is  to  be  based  on  a  complex  analysis,  some  understanding  of 
the  assumptions,  the  relationships,  and  the  sensitivity  of  certain  parameters 
must  be  transmitted  from  the  study  to  the  decision  maker  without  the  deci- 
sion maker  having  to  re-create  the  complete  analysis  of  the  systems  analyst. 
This  is  particularly  true  when  the  problem  involves  multiple  alternatives 
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and  multiple  criteria  of  effectiveness.   In  these  cases  flexible  methods  of 
presentation,  which  allow  the  decision  maker  to  contribute  his  own  value 
judgements,  must  be  employed. 

The  third  (3)  interface  is  between  the  analyst  and  the  study.   As  data 
becomes  more  voluminous  and  solutions  require  greater  compuation, the  analyst 
must  construct  his  models  and  carry  out  the  study  with  the  aid  of  the  computer. 
His  interactions  with  the  study  are  through  a  terminal  or  through  a  computa- 
tion center.   As  a  consequence,  he  is  concerned  with  such  things  as  turnaround 
times,  program  formats,  program  diagnostic  capability.   Improving  these  inter- 
faces are  important  in  improving  the  efficiency  of  the  analyst  and  the  quality 
of  the  final  study. 

l^e  fourth  interface  (4)  is  between  the  analyst  and  the  organization's 
stock  of  experience,  i.e.,  the  bank  of  previous  studies  and  models  and  the 
bank  of  current  data.   As  multiple  agencies  and  organizations  engage  in 
the  preparation  of  cost-effectiveness  studies,  experience  is  developed  with 
potentially  wide  application.   If,  however,  methods  for  collecting  this 
information  and  disseminating  it  to  the  other  analysists  are  not  developed, 
no  long-term  learning  will  develop  in  the  study  process.   Thus,  attention 
must  be  given  to  the  interface  between  the  analyst  and  the  existing  store 
of  knowledge. 

The  fifth  interface  (5)  is  between  the  decision  maker  and  the  study  and 
data  bank.   Frequently  the  decision  maker  will  wish  to  compare  the  results, 
assumptions,  and  approach  of  a  current  study  with  those  of  some  previous 
studies.   Existence  of  a  data  file  and  ability  to  search  it  first  by 
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report  titles  and  then  by  abstracts  will  be  necessary  for  such  a  comparison 
process.   From  the  information  drawn  from  the  storehouse  of  previous 
knowledge  he  can  ensure  the  latest  findings  and  approaches  are  brought  to 
bear  on  each  study. 

Findings 

The  findings  which  we  will  discuss  here  relate  primarily  to  the  inter- 
face problems  just  described.   They  are  presented  here,  followed  by  the 
factors  which  led  us  to  believe  that  they  were  important  and  constructive. 

In  reviewing  these  findings^  the  reader  should  keep  in  mind  the  limita- 
tions of  the  sample  size  of  twenty  studies.   Statistical  and  numerical 
inferences  should  be  considered  only  as  suggestive  of  trends.   To  interpret 
these  findings,  one  should  also  consider  the  specific  studies  which  were 
included  in  the  sample  since  they  establish  the  representativeness  of  the 
study. 

Finding  1 

In  analyzing  the  decision-maker-analyst  interface,  it  was  found  that  the 
decision  maker's  participation  in  the  study  process  would  appear  to  be  more 
effectively  allocated  if  a  greater  proportion  of  his  time  were  concentrated 
in  the  early  phases  of  the  study.   At  the  present  time,  the  decision  maker's 
attention  is  focused  almost  exclusively  on  the  final  report,  the  very  end 
of  the  study.   By  this  time,  the  study  structure  has  often  become  rigid   and 
cannot  easily  be  revised  to  reflect  the  particular  interests  of  the  decision 
maker.   To  avoid  this  fait  a  complit,  it  would  appear  more  efficacious  if 
the  decision  maker  considered  with  the  analyst  the  objectives  of  the  study 
and  the  preliminary  proposal  for  performing  this  study. 


:..  , 
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There  were  five  basic  points  which  led  us  to  this  conclusion.   First,  at 
least  half  the  studies  which  we  examined  had  to  be  re-done  after  the  review 
of  the  final  report  by  the  decision  maker.   As  a  consequence,  the  time  spent 
by  the  decision  maker  on  these  studies  was  at  least  doubled  because  he  had 
to  review  two  or  more  final  reports.   In  several  cases,  there  were  more  than 
two  reviews.   It  would  seem  that,  if  the  decision  maker  could  participate 
earlier  in  the  proposal,  or  study  execution  process,  some  of  these  reviews 
might  be  avoided.   There  is  even  a  potential  for  a  decrease  in  the  decision 
maker's  participation  time  as  well  as  an  increase  in  the  general  quality  of 
the  studies. 

The  second  point  which  led  us  to  believe  that  earlier  participation  by 
the  decision  maker  might  be  more  effective  was  that  studies  were  of  higher 
quality  in  which  there  was  some  form  of  proposal  review,  regardless  of 
whether  it  was  a  decision  maker  or  his  immediate  representative.   The 
criteria  of  quality  is  that  which  is  referred  to  and  described  in  the  sec- 
tion on  criteria  of  effectiveness. 

The  third  point  was  that  the  principal  dissatisfaction  expressed  by  per- 
sonnel at  the  decision  maker  level  was  that  studieswere  not  responsive  to 
the  problems  which  they  were  facing.   In  particular,  the  questions  which 
were  sometimes  answered  by  the  studies  were  not  the  questions  which  the 
decision  maker  wished  to  address.   Further,  the  information  required  for 
some  decisions  was  not  always  developed  in  the  study  process.   It  was  this 
concern  which  led  to  the  initiation  of  this  study. 
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The  fourth  consideration  was  that,  for  those  studies  which  had  to  be 
reviewed  or  re-done,  over  70  per  cent  of  the  reasons  given  for  revision 
were  due  to  factors  related  to  the  decision  maker's  judgement  rather  than 
to  methodological  problems.   In  general,  there  were  very  few  studies 
re-done  or  revised  because  a  computational  or  methodological  error  was 
found,  or  because  new  data  was  introduced.   In  most  of  the  cases,  the 
reason  for  revision  was  because  the  decision  maker  desired  other  conditions 
to  be  examined,  other  alternatives  to  be  examined,  or  new  interpretations 
of  the  results  to  be  made.   Some  of  these  evaluative  revisions  could 
probably  be  identified  earlier  in  the  study  process,  thus  avoiding  the 
wasted  or  inefficient  effort  which  goes  into  the  revision  process. 

Fifth  and  finally,  the  late  intervention  by  the  decision  maker  appears 
to  be  introducing  an  effect  %rtiich  the  decision  maker  himself  felt  was  un- 
desirable.  In  particular,  at  the  analyst  level,  there  is  a  general  feeling 
that  higher  headquarters  desire  each  study  to  be  more  sophisticated,  more 
complex,  and  employ  a  newer  methodological  approach  than  the  previous  study. 
On  the  other  hand,  our  discussions  with  headquarters  decision  maker 
representatives  revealed  a  desire  for  simpler  less  complex  models.   The 
reason  for  this  misunderstanding  appears  to  be  that  the  decision  maker 
desires  a  greater  generality  or  ability  to  respond  to  "what-if"  questions 
while  the  analyst  anticipating  many  "what  if"  questions  had  tried  to 
provide  built-in  answers,  thus  increasing  complexity  and  sophistication. 
Earlier  participation  by  the  decision  maker,  therefore,  might  lead  to  a 
greater  mutual  understanding  of  the  level  of  analysis  which  is  appropriate 
to  each  problem. 
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Findtng  2 

Again,  with  respect  to  the  decision-maker-analyst  interface,  great  at- 
tention should  be  devoted  to  the  definition  of  the  decision  which  the  deci- 
sion maker  must  ultimately  make.   While  increased  emphasis  has  been  given 
to  identifying  objectives  and  specifying  the  problem,  there  do  appear  to 
have  been  a  number  of  instances  in  which  the  decisions  to  be  made  still 
were  not  either  clearly  understood  or  if  understood,  not  adequately  in- 
corporated into  the  study.   The  definition  of  the  decision  must  be  stated  in 
operational  terms  so  that  it  is  meaningful  to  the  analyst.   It  is  a  mistake 
I   to  say  that  non-responsive  studies  are  always  the  fault  of  the  analyst.   They 
may,  in  fact,  be  the  fault  of  the  decision  maker  for  not  accurately  specifying 
exactly  what  decision  he  must  make  and  for  relying  on  the  analyst  to  develop, 
not  only  the  information  required  for  the  decision,  but  to  develop  the  deci- 
sion at  hand  as  well. 

Several  factors  led  us  to  believe  that  this  was  an  important  area.   First, 
on  the  average,  four  separate  distinguishable  organizations   review  and  inter- 
pret the  study  objectives  before  they  reach  the  study  analyst.   It  is  not 
surprizing,  therefore,  that  by  the  time  the  analyst  receives  these  objectives 
they  do  not  necessarily  reflect  the  decision  needs  of  the  ultimate  user  of 
the  study.   The  organizational  distance  between  the  study  analyst  and  the 
decision  maker  is  extremely  long. 

A  further  problem  in  aligning  the  study  with  the  needs  of  the  decision 
maker  is  that  in  general,  at  least  two,  and  as  many  as  five  organizations 
have  responsibility  for  the  preparation  of  the  study   Each  brings  a  dif- 
ferent point  of  view  to  the  same  problem.   In  most  cases   the  organizations 
performing  the  study  are  equally  distant  organizationally  from  the  decision 
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maker .   This  makes  a  ready  territory  for  the  compromise  of  biases  at  the 
sacrifice  of  the  decision  maker's  real  goals  and  value  judgements  as  they 
may  have  originally  been  expressed  in  the  study  objective. 

Third,  there  are  a  similar  number  of  intermediary  organizations  which 
review  and  revise  the  results  of  the  study.   These  again  produce  a  filter 
between  the  study  analyst  and  the  decision  maker,  with  respect  to  the  inter- 
pretation of  results  found  in  the  study  analysis. 

Fourth,  it  is  the  general  impression  of  the  study  analysts  that  they 
make  a  number  of  calls  and  hold  copious  conversations  with  people  at  the 
decision  maker  level,  so  that  objectives  are  adequately  interpreted.   On  the 
other  hand,  discussions  with  decision  makers  and  members  of  their  staff 
reveal  that  they  do  not  believe  that  analysts  do  adequately  communicate  with 
the  decision  maker.   This  misunderstanding  would  appear  to  be  indicative  of 
the  communication  problems  that  exist  between  the  analyst  and  the  decision 
maker . 

Finding  3 

The  final  consideration  in  the  decision-maker  analyst  interface  area 
concerns  the  facilitation  of  communication  between  the  two. 

The  policy  oriented  decision  maker  appears  to  have  some  difficulty,  as 
might  be  expected,  in  discussing  issues  with  the  technically  oriented 
analyst.   This  communication  issue  is  not  only  a  matter  of  language  and 
semantics  but  also  a  matter  of  the  outlook  and  background  of  the  personnel 
involved.   Increased  time  on  the  part  of  the  decision  maker  in  defining  his 
problem  early  in  the  model  formulation  process  would  undoubtedly  help.   In 
the  cost-effectiveness  studies  that  we  examined,  it  was  noted  that  the 
model  did  not  form  an  effective  communications  medium  between  the  analyst 
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and  the  decision  maker.   One  way  of  helping  with  the  communications  problem 
would  be  to  have  both  parties  talking  around  some  common  structure  of  some 
specific  variables.   That  is,  the  semantic  problem  is  reduced  as  conversa- 
tions become  more  and  more  specific.   However,  the  technology  and  the  model 
methodology  that  is  currently  being  used  in  the  studies  that  we  examined  is 
not  such  as  to  provide  this  communications  vehicle.   The  models  were  all 
extremely  detailed,  and  the  technology  did  not  allow  testing  of  even  portions 
of  the  model  in  any  flexible  way.   As  a  natural  outcome  of  this,  the  review 
committees  that  sit  and  examine  the  methodology  and  other  issues  surround- 
ing the  technical  aspects  of  the  model  are  not  able  to  provide  a  decision 
maker  with  an  intermediate  source  of  review.   That  is, a  technical  review 
progress  committee  has  to  be  technically  oriented  if  they  hope  to  understand 
and  comment  effectively  on  the  technology  involved  in  each  of  the  models. 
This  group,  however,  has  exactly  the  same  problems  in  communicating  with  the 
decision  maker  as  the  analysts  themselves. 

Finding  4 

Turning  now  to  the  interface  between  the  decision  maker  and  the  study^ 
we  found  that  there  is  little  consistancy  in  the  presentation  of  study 
results.   At  the  present  time,  the  decision  maker  must  read  between  ten  and 
thirty  studies  a  year   Consequently,  with  this  expenditure  of  time  in  a 
repetitive  activity,  it  becomes  important  to  examine  the  factors  which  af- 
fect his  ability  to  comprehend  and  make  meaningful  decisions  from  study 
reports  with  the  minimum  expenditure  of  time.   The  development  of  a  minimum 
standard  set  of  components  with  common  definitions  of  terms  would  signi- 
ficantly decrease  the  time  required  for  the  decision  maker's  evaluation  of 
the  study. 
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The  factors  which  led  us  to  this  conclusion  were  that  a  review  of  the 
twenty  studies  which  we  evaluated  revealed  a  radical  difference  in  the 
terminology  employed  and  in  the  report  organization  structure.   There  was 
almost  no  similarity  between  tables  of  contents.   Further,  similar  words  in 
different  tables  of  content  designated  different  elements  of  information. 
Under  such  circumstances,  it  is  obvious  that  the  decision  maker  cannot 
quickly  go  to  the  table  of  contents,  search  out,  for  example,  the  assumptions, 
the  conclusions,  criteria  of  effectiveness  and  find  the  information  which 
he  seeks. 

The  second  point  arose  from  the  examination  of  a  set  of  25  information 
components  in  each  study  which  we  felt  were  necessary  for  ease  of  comprehen- 
sion.  These  components  were: 

(1)  the  documentation  which  led  to  the  initiation  of  the  study 

(2)  the  identification  and  description  of  the  study  team  and  its 
organizational  relationships 

(3)  the  cost  of  the  study 

(4)  planned  and  actual  schedule  for  the  study 

(5)  the  definition  of  the  decision  to  be  made  by  the  decision 
maker 

(6)  a  statement  of  the  study's  objectives 

(7)  ground  rules  and  assumptions  employed  in  the  study 

(8)  the  alternatives  considered  in  the  study 

(9)  the  criteria  of  effectiveness  employed  in  the  study 

(10)  the  sources  of  data  used  in  the  study 

(11)  the  prior  studies  relevant  to  the  study 
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(12)  the  approach  used  in  the  treatment  of  risk  and  uncertainty 

(13)  the  limitations  and  the  exclusions  of  the  methodology 

(14)  the  dimensions  of  the  cost  breakout 

(15)  a  flow  chart  of  the  model  used  in  the  study 

(16)  a  description  of  the  model  used  in  the  study 

(17)  the  major  parameters  employed  in  the  model 

(18)  the  structure  of  the  integration  of  the  multiple  measures 
of  effectiveness 

(19)  the  structure  for  the  integration  of  cost  with  the  multiple 
measures  of  effectiveness 

(20)  the  major  sensitivity  and  contingency  analyses  carried  out 

(21)  the  relationship  of  the  analysis  results  to  the  decision  to 
be  made 

(22)  answers  to  the  specific  questions  posed  by  the  decision 
maker 

(23)  a  matrix  of  the  multiple  measures  of  effectiveness  versus 
the  alternatives 

(24)  recommendations  for  methodological  improvement 

(25)  an  analysis  of  the  specific  qualitative  factors  not  covered 
in  the  quantitative  analysis. 

Our  analysis  of  the  ease  of  finding  each  of  these  elements  in  the  dif- 
ferent studies  showed  that  in  less  than  half  of  the  studies  were  each  of 
these  elements  easily  found.   Unfortunately,  there  was  little  consistency 
in  the  ability  to  find  all  of  the  elements  in  any  one  particular  study. 
Rather,  studies  tended  to  have  half  of  the  elements  well  presented   and  the 
other  half  poorly  presented.   The  clear  consequence  of  this  is  that  the 
decision  maker  must  read  each  study  as  new  and  entirely  different  from  all 
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the  rest  which  he  has  read.   He  must  read  it  in  its  entirety  to  be  sure  first^ 
that  he  gains  all  of  the  information  present   and  secondly,  to  find  out  what 
information  is  not  contained  in  the  report.   A  standard  set  of  components 
uniformly  understood  and  presented  might  significantly  reduce  the  decision 
maker's  time  input  to  each  study. 

FindinR  5 

The  decision  maker's  interaction  with  the  cost-effectiveness  model  is 
limited,  at  the  moment, to  printed  material  and  verbal  exchanges  between  the 
analysts  and  the  decision  makers  involved.   The  models  are  typically,  extremely 
complex,  and  the  printed  page  does  not  do  justice  to  the  kinds  of  relation- 
ships and  assumptions  that  have  been  built  into  the  model  nor  indeed  for 
the  implications  of  these  assumptions  on  the  decision  at  hand.   Quite  apart 
from  the  variation  in  the  presentation  and  preparation  of  the  reports  which 
is  discussed  above,  there  is  the  issue  of  static  versus  dynamic  information 
presentation.   Certain  concepts  and  assumptions  are  not  easily  understood 
unless  the  decision  maker  is  able  to  see  some  time  trace  of  the  variables 
involved  or  has  been  actively  involved  in  the  construction  of  the  model 
itself.   Since  the  decision  makers  have  very  limited  time,  it  is  not  surpris- 
ing to  find  no  instances  of  active  decision-maker  involvement  in  the  model 
construction  process.   However,  this  does  imply  that  the  decision  makers  are 
not  really  conversant  with  the  model  itself  at  the  moment  of  decision. 

Similarly,  the  structure  of  the  models  which  we  examined  were  designed 
to  implement  the  analyst's  notion  of  the  appropriate  analysis  for  solving 
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the  problem  as  he  perceived  it.   Due  to  constraints  in  time  and  budget,  the 
analysts  did  not  appear  to  have  time  to  develop  parameterized  models  for  the 
decision  maker's  use.   The  parameters  that  were  available  for  the  decision 
maker  to  change,  typically  did  not  reflect  the  kinds  of  parameters  he 
wanted  to  change,  and  the  result  was  that  the  study  had  to  be  re-done, 
rather  than  simply  re-run  with  different  data.   Similarly,  the  decision 
maker  did  not  have  the  ability  in  any  of  the  cases  we  examined,  to  alter 
the  structure  of  the  model   itself  without  the  necessity  of  going  through 
the  whole  rebuilding  of  the  model.   The  decision  maker  did  not  have  the 
ability  to  modify  the  parameters  of  the  model  with  a  given  structure,  nor 
did  he  have  the  ability  to  modify  the  structure  itself.   In  many  of  these 
very  complex  models,  this  ability  may  not  realistically  be  available  on 
any  large  scale,  but  the  studies,  decision  makers, and  analysts  involved  in 
the  cases  to  which  we  had  access,  did  not  identify  to  the  researchers  any 
concern  with  this  problem  or  any  notion  that  there  could  be  some  marginal 
improvement  on  these  issues. 

Finding  6 

The  third  interface  was  between  the  analyst  and  the  study.   In  this  area, 
we  found  that  many  of  the  models  were  written  in  different  languages  to  run 
on  different  computers.   These  models  were  often  incompatible  with  data 
structures  and  data  bases  which  had  been  set  up  from  previous  studies. 
All  of  the  equipment  involved  runs  in  a  batch  mode  which  provides  a  certain 
amount  of  efficiency  for  the  machine  but  little  or  no  efficiency  for  the 
analyst  himself.   The  scarce  resource  in  all  studies  was  the  time  of  the 
qualified  analysts,  and  it  would  seem  appropriate  to  organize  resources  to 
keep  these  key  people  usefully  busy  rather  than  have  them  spend  a  great  deal 
of  time  on  mechanical  functions. 
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Finding  7 

Further  observations  relating  to  the  analyst-study  interface  were  that 
there  is  little .clearly  identified  slack  time.   Our  observations  were  that, 
although  the  number  of  systems  analysis  organizations  with  the  AMC  is  growing, 
the  capability  for  systems  analysis  is  still  a  scarce  resource.   Consequently, 
there  are  significant  demands  placed  on  existing  in-house  organizations. 
These  demands  are  of  two  types:   First,  there  is  a  heavy  demand  to  perform 
or  participate  in  a  large  number  of  studies.   Secondly,  the  time  period 
allowed  for  preparation  of  any  particular  study  is  very  short.   This  leads 
to  the  use  of  those  analytical  approaches  which  have  most  recently  been 
employed  locally  and  are  immediately  available  to  them.   Unfortunately, 
this  constrains  innovation  in  methodology.   Consequently,  some  period  of 
time  might  beneficially  be  set  aside  for  devotion  strictly  to  innovation  in 
methodology. 

A  further  point  is  that  the  methodology  is  not  tied  to  the  study  fund- 
ing decision.   Consequently,  there  is  little  attention  paid  to  the  level  of 
analysis  which  is  most  efficient  for  the  problem  at  hand.   The  rule  of 
thumb  which  appears  to  be  used  is  to  use  the  most  recently  developed 
technique  that  the  analyst  knows,  with  a  slight  improvement.   Rarely  are 
radical  alternatives  considered. 

This  lack  of  free  time  for  innovation,  therefore,  is  felt  not  to  be  the 
fault  of  the  analyst  but  the  fault  of  the  system  which  does  not  present 
opportunity  or  incentive  to  make  efficient  time  allocation  decisions. 
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Flnding  8 

With  respect  to  the  interface  between  the  analyst  and  the  stock  of  ex- 
perience from  previously  constructed  models  and  collected  data,  we  found 
that  the  analysts  were  often  not  familiar  with  the  work  that  had  been  done 
in  other  groups,  and  this  resulted  in  a  considerable  amount  of  effort  to 
construct  submodels  which  were  already  in  existence.   There  did  not  seem 
to  be  any  organized  flow  of  information  or  any  central  clearing  house  to 
act  as  a  focal  point  for  methodology,  or  data  basis  models,  or  computer 
technology. 

Each  study  represents  a  considerable  investment  not  only  in  funding,  but 
in  the  development  of  new  data  and  in  the  employment  of  scarce  talent.   It 
was  a  rare  instance  that  we  found  a  study  which  used  or  employed  significant 
portions  of  another  study.   This  did,  of  course,  occur  but  not  to  the  extent 
that  might  be  possible  if  greater  emphasis  were  placed  on  this  approach. 
Not  only  were  studies  rarely  drawing  upon  resources  already  developed  in 
other  studies,  but  there  appeared  to  be  few  mechanisms  within  the  AMC 
study  activity  which  were  specifically  directed  toward  the  achievement  of 
transferability.   In  particular,  there  were  no  universally  known  repositories 
of  data  on  either  cost,  performance,  or  methodology.   Further,  there  were 
few  computer  tie-ups  between  system  analysis  organizations  which  would  enable 
tliem  to  experiment  freely  with  each  other's  system  models  without  the  high 
cost  of  travelling  back  and  forth  physically  for  this  purpose.   Further, 
there  were  no  provisions  within  the  AMC  for  updating  models  with  respect 
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to  new  technological  forecasts,  new  threat  forecasts,  new  cost  data,  and  other 
factors.   Yet,  it  was  clear  that  some  of  these  models,  particularly  the  weapon 
class  models,  such  as  Reval  Wheels,  SAWS,  etc.  had  significant  potential 
for  later  application. 
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Re commendations 

In  this  section,  we  will  present  some  recommendations  which  we  believe 
appropriate  in  light  of  the  findings  just  presented. 

Recommendation  1 

Our  first  recommendation  is  that  the  present  study  process  should  be  some- 
what modified.   The  current  study  generation  process  is  described  in  Figure  1. 
While  there  are  numerous  exceptions  to  this  chart,  it  depicts  fairly  well  the 
decision  points  in  the  study  process.   Figure  2  is  an  alternative  approach 
to  this  process  in  which  we  have  tried  to  make  some  changes  based  on  our 
findings . 

The  first  suggested  change  deals  with  the  interface  between  the  decision 
maker  and  the  analyst.   It  is  recommended  that  a  decision  definition  phase 
be  created  in  which  greater  attention  is  paid  to  the  particular  decisions 
which  must  be  made  and  to  the  type  of  study  which  is  most  relevant  to  the 
decision  at  hand.   We  have  distinguished  two  types  of  activities  in  this 
phase;  first,  the  definition  of  the  decision  to  be  made  and  second,  the 
definition  of  the  study  requirements.   At  present,  it  is  our  impression  that 
these  two  are  subsummed  in  an  activity  called  development  of  study  objectives 
which,  perhaps,  is  inadequate   by  itself. 

The  second  recommended  change,  which  also  affects  the  decision-maker 
analyst  interface  is  increasing  the  importance  of  early  participation  by 
the  decision  maker  in  the  study  process.   We  have  reduced  the  importance 
of  his  participation  at  the  end  of  the  study  and  emphasized  his  participa- 
tion in  the  "command  review"  of  the  proposal  and  in  the  "evaluation  of  the 
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preliminary  conclusions".   Each  of  these  reviews  should  facilitate  the  in- 
corporation of  his  views  and  value  judgements  at  points  in  the  study  process 
where  they  can  be  more  effectively  and  efficiently  accommodated. 

The  third  change,  having  impact  on  the  analyst-study  interface, is  in  the 
procedure  for  determining  the  level  of  analysis  and  funding  appropriate  to 
the  study.   Under  the  current  system,  there  is  no  objective  determination 
of  the  funding  level  appropriate  to  each  study.   Rather,  due  to  the  shortage 
of  time  and  the  failure  to  explicitly  recognize  these  as  important  decisions, 
the  level  of  analysis  and  the  funding  decision  are  determined  solely  on  the 
basis  of  the  availability  of  resources  and  the  level  of  funding  used  on  the 
most  recent  study  of  a  similar  nature.   We  believe  that  funding  decisions 
should  be  specifically  geared  to  the  level  of  the  necessary  analysis.   To 
achieve  this, consideration  might  be  given  to  providing  study  funds  for  a 
study  by  providing  annual  blocks  of  funds  to  the  study  organizations  which 
is  the  approach  used  frequently  now.   Block  grants  have  their  place,  but 
their  place  is  not  in  the  conduct  of  individual  studies   but  rather  in  the 
undertaking  of  central  activities  such  as  methodology  improvement,  data 
storage,  and  other  functions  to  be  discussed  shortly. 

The  final  change  relating  to  the  analyst-study  and  data  bank  interface 
appears  in  the  implementation  phase,  in  which  it  is  explicitly  recognized  in 
the  "alternative  system",  that  studies  have  application  beyond  the  point  of 
initial  decision.   They,  therefore,  must  be  stored  and  updated  and  re-evaluated 
so  that  they  are  suitable  for  use  in  later  decisions  and  in  implementation 
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adjustment  programs.   These  particular  changes,  we  believe,  will  help  signi- 
ficantly in  making  the  study  responsive  to  the  decision  maker's  needs   and 
in  reducing  their  cost  and  complexity. 

Recommendation  2 
Introduction 


A  basic  premise  of  formal  models  as  part  of  the  decision  making  process  is 
the  notion  that  the  combination  of  the  model  and  the  decision  maker  is  more 
powerful  than  either  technique  alone.   Decision  makers  unaided  by  formal 
analysis  and  rigorous  consistent  data  evaluation  are  unlikely  to  be  as  ef- 
fective as  they  could  be  with  a  solid  underpinning  of  fact, as  well  as  as- 
sumption and  implication  testing.   Similarly,  we  have  yet  to  build  formal 
models  which  are  capable  of  making  intelligent  decisions  without  the  con- 
siderable assistance  of  managerial  talent.   The  design  goal,  then,  is  to 
mix  both  resources,  the  manager,  and  the  formal  models  in  proportions  such 
that  we  can  employ  the  relative  advantages  of  each.   The  decision  maker  can 
provide  his  judgement,  his  ability  to  structure  unorganized  material,  and 
his  considerable  ability  in  recognizing  patterns  and  drawing  on  previous 
experience.   The  machine  and  the  formal  models,  on  the  other  hand,  have 
considerable  computation  ability.   They  exhibit  a  great  deal  more  consistency 
than  the  human  decision  maker  and  are  able  to  draw  on  the  vast  data  base, 
collecting  detailed  facts  which  are  unavailable  to  the  normal  human.   This 
mix  of  skills  has  been  discussed  in  the  planning  and  control  literature  in 
some  depth,  and  we  would  assert  that  the  technology  has  finally  got  to  the 
point  where  such  combinations  are  economically  feasible  and  have  considerable 
impact  on  complex  decision  processes. 
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Research  in  complex  industrial  decision  making  has  shown  that  such  man- 
machine  combinations  have  a  considerable  impact  on  the  decision  processes 
of  higher  level  managers.   In  summary,  this  impact  could  be  described  under 
five  headings: 

1)  Time.   The  time  actually  spent  in  the  decision  process,  and  the 
elapsed  time  between  the  starting  of  the  problem  solving  process   and  the 

I   final  solution  is  considerably  reduced  with  the  use  of  such  an  inter- 
active man-machine  system.   In  the  experiments  conducted  to  date,  an 
order  of  ten  to  one  reduction  in  both  decision  making  and  elapsed  time 
has  been  exhibited.   This  time  is  not  only  significant  itself,  but  it  also 
permits  a  different  form  of  decision  making  to  be  used.   It  allows  the 
decision  maker  to  move  from  a  serial  process  with  considerable  inter- 
ruptions for  implication  testing   to  an  interactive  parallel  process  of 
solving  problems. 

2)  Alternative  Testing.   This  parallel  form  of  decision  making  allows 
the  manager  to  test  alternatives  and  see  their  implications  over  time. 
That  is,  he  can  ask  "what  it"  questions,  and  the  system  can  execute  their 
implications  and  provide  him  with  output  variables  that  show  the  change  in 
his  solution 

3)  Several  Solutions.   This  testing  of  various  alternatives   results 
in  the  decision  making  having  more  viable  solutions  from  which  to  make  his 
final  choice.   That  is,  having  tested  a  number  of  alternatives,  he  can  begin 
to  develop  a  feel  for  the  magnitude  and  dynamics  of  the  system  and  can  then 
use  his  creativity  and  vast  experience  to  identify  other,  perhaps  better, 
solutions . 
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4)  Criteria  Development.   As  a  result  of  this  interaction  with  the  model 
and  its  structure,  the  decision  maker  begins  to  develop  some  criteria  that 
enable  him  to  understand  the  interactions  among  this  system  more  clearly. 
This  basic  systems  comprehension  not  only  improves  his  understanding  of 
likely  alternatives,  but  also  gives  him  a  mechanism  that  forces  some  rigor 
and  consistency  into  his  decision  process. 

These  four  kinds  of  impacts,  briefly  discussed  above,  are  indicative  of 
the  sorts  of  benefits  that  are  possible  with  the  use  of  the  newer  technology 
in  the  form  of  interactive  display  systems.   Such  systems  will  not  solve  all 
the  decision  maker  -  model-  analyst  interface  problems,  but  will  alleviate 
some  of  them  and  more  importantly  perhaps,  provide  a  mechanism  that  can 
lead  to  iterative  improvement  over  time.   The  interactive  display  system 
provides  a  mechanism  to  trace  the  decision  process  over  time,  and,  with 
research  effort  devoted  to  analyzing  such  traces,  it  becomes  possible  to 
formalize  the  improvement  of  the  decision  making  process.   This  latter 
feature  suggests  that, in  the  long  term,  it  will  be  feasible  to  use  such 
traces  as  raw  data  from  which  we  can  develop  an  understanding  of  the  pat- 
terns inherent  in  a  decision  process.   Given  an  understanding,  it  may  then 
be  possible  to  develop  heuristics  which  will  allow  the  machine  to  become  an 
active  member  in  the  problem  solving  process. 

As  a  first  step  toward  such  an  ultimate  solution,  we  would  recommend  that 
research  be  started  with  a  view  to  developing  a  problem  presentation  package 
on  an  interactive  terminal.   That  is,  it  would  be  desirable  to  present  to  the 
decision  maker  the  results  of  the  cost-effectiveness  study  on  the  interactive 
terminal.   This  would  not  only  allow  the  use  of  dynamic  displays,  where  the 
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decision  maker  could  see  variables  Interacting  over  time,  but  would  also  al- 
low the  use  of  graphical  presentation  and  other  more  creative  forms  of  com- 
munication.  In  addition,  of  course,  the  crucial  aspect  of  the  display  would 
be  to  allow  the  manager  to  request  data  and  results  presentations  in  the  way 
that  appeals  to  him  for  a  particular  decision.   Given  a  particular  context 
and  his  understanding  of  the  decision  that  has  to  be  made,  he  could  use  his 
own  judgement  to  view  the  results  and  to  choose  which  variables  are  to  be 
contrasted,  and  in  what  sequence.   This  essentially  allows  the  problem 
finding  process  and  the  model  testing  process  to  be  under  the  control  of 
the  decision  maker  involved. 

Similarly,  it  would  be  possible  for  the  decision  maker  to  test  parameters 
and  make  changes  in  the  model  structure  through  the  use  of  such  a  terminal. 
He  would  then  have  quick  response  to  many  of  the  changes  he  would  request,  and 
this  would  lead  him  on  an  Interactive  path  through  the  assumptions  and 
structure  of  the  model.   In  such  a  way,  in  a  one  or  two-hour  working  session, 
he  could  develop  a  precise  feel  for  the  power  and  limitations  of  the  model 
involved.   This  would  in  turn  allow  him  to  base  his  decision  on  a  much 
higher  level  understanding  of  the  model  than  he  currently  has. 

This  interactive  capability  not  only  permits  him  to  look  at  the  results 
of  the  model  in  a  dynamic  and  flexible  fashion,  but  also  to  test  for  himself 
the  sensitivity  of  the  model  and  make  the  kind  of  changes  he  feels  are 
relevant,  given  the  decision  that  he  is  faced  with,  and  his  understanding  of 
the  political  environment. 
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Re commendation  3 

As  a  further  improvanent  of  the  decision-maker  study  interface,  it  is 
our  recommendation  that  a  minimum  set  of  standard  elements  be  selected  and 
required  for  each  study.   As  pointed  out  earlier,  to  conserve  the  time  ex- 
pended by  the  decision  maker,  some  standardization  in  the  report  format  is 
necessary.   The  standard  elements  listed  in  the  findings  might  be  an  ap- 
propriate minimum  set.   Some  of  these  elements  we  did  not  find  in  the  studies 
we  examined;  for  example,  source  of  the  data,  cost  of  the  study,  original 
plan  time  and  actual  time  to  perform  the  study.   However,  these  we  believe 
are  essential  for  accurate  evaluation  of  the  study  process  and  will  have 
beneficial  effects  beyond  a  simple  transfer  of  information  to  the  decision 
maker. 

Recommendation  4 


The  next  two  recommendations  deal  with  the  interface  between  the  analyst 
and  the  store  of  existing  knowledge. 

In  the  findings,  we  identified  the  importance  of  increasing  the  trans- 
ferability of  the  benefits  associated  with  the  studies.   To  achieve  this, 
we  recommend  the  creation  of  an  organization  which  will  be  responsible  for 
insuring  and  achieving  this  transferability.   Specifically,  it  will  perform 
the  following  functions. 

1)  It  will  co-ordinate  the  data  bank  activities  and  insure  that 
at  some  known  point  within  AMC  the  latest  data  for  each 
weapon  and  environmental  condition  are  kept  and  updated. 
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2)  It  will  co-ordinate  a  model  bank  and  insure  that  those  models 
which  have  later  relevance  are  kept  up  to  date  as  new  data 

is  supplied  to  the  data  bank. 

3)  It  will  insure  that  methodological  improvement  programs  are 
undertaken  by  each  study  group  and  will  be  responsible  for 
the  approval  and  funding  of  methodological  improvement 
programs . 

4)  This  group  will  be  responsible  for  co-ordinating  the  AMC  and 
CDC  study  activities. 

This  is  not  a  recommendation  to  establish  a  centralized  data  bank  and  a 

centralized  model  storage  facility  which  will  perform  all  the  methodology 

improvement  programs  itself.   The  importance  of  decentralization  seems 

evident  by  virtue  of  the  fact  that  specialized  expertise  exists  in  many  of 

the  agencies.   It  is  clear  that  air  defense  expertise  is  presently  at 

Huntsville,  that  artillery  expertise  is  presently  at  BRL,  that  mobility 

expertise  is  presently  at  the  Tank  Command  and  in  other  places.   However, 

with  this  level  of  expertise  so  spread  around  geographically  it  is  important 

to  have  a  group  which  provides  certain  central  co-ordinating  functions  to 

these  multiple  activities.   In  particular,  this  central  group  should  set 

standards  for  reports,  model  structure,  data  format,  presentation,  etc.   It 

should  audit  the  data  bank  activities,  the  model  storage  activities,  and 

methdology  improvement  activities  at  each  of  the  study  centers.   It  should 

allocate  funding  to  each  of  these  study  centers  in  accordance  with  their 

needs  for  data  bank  maintenance,  model  storage  maintenance  and  methodological 
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improvement.   Finally,  it  should  serve  as  a  central  point  or  contact  to  which 
all  persons  may  go  when  desiring  information  about  the  existence  of  data 
models,  methodology,  or  other  points  of  interest  within  the  AMC  study 
system.   At  present,  there  appears  to  be  a  considerable  lack  of  under- 
standing of  the  capabilities  which  are  existent  throughout  the  system. 

I 

While  this  information  is  understood  at  the  central  headquarters  level,  it 

is  not  well  understood  within  the  study  analysis  groups.   We  have  suggested, 
therefore,  the  centalization  of  certain  control  activities,  but  the  main- 
tenance of  the  decentralization  of  the  major  operational  activities.   In 
short,  we  are  suggesting  an  information  retrieval  facility  where  the  analyst 
can  inquire  on  a  remote  basis  about  the  characteristics  of  data  available 
in  a  central  data  bank  or  the  characteristics  of  models  available  in  a 
central  model  bank.   Such  a  file,  of  course,  could  also  be  used  by  deci- 
sion makers  or  decision-maker  staff  in  deciding  the  funds  and  time  neces- 
sary to  make  a  study  on  a  particular  system.   In  fact,  such  a  central  bank 
could  also  have  presentation  devices  and  software  packages  to  allow  the 
dynamic  presentation  of  material  to  decision  makers.   In  any  event,  the 
costs  and  conceptual  effort  involved  in  creation  of  such  a  facility  is 
relatively  small  in  light  of  the  benefits  this  would  provide,  and  the 
reduction  in  time  and  analyst  effort  needed  for  these  complex  studies. 

Recommendation  5 

In  order  to  achieve  the  kinds  of  control  and  transferability  of  benefits 
cited  earlier,  certain  kinds  of  standardization  will  be  necessary.   One  will 
be  a  standardization  of  the  classes  of  models.   For  example: 
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(1)  There  are  parametric  analyses  for  the  purpose  of  establishing 
optimal  weapon  designs. 

(2)  There  are  utility  evaluation  studies  for  the  comparison  of 
one  weapon  approach  against  another. 

(3)  There  are  class  analysis  tudies  for  the  purpose  of  evaluat- 
ing entire  classes  of  weapons  such  as  small  arms,  trucks, 
etc. 

(4)  There  are  product  improvement  studies  for  evaluating  modifi- 
cations to  existing  weapons. 

(5)  There  are  requirements  analyses  for  evaluating  the  environ- 
ment in  order  to  specify  the  optimal  weapon. 

(6)  There  are  effectiveness  studies  for  simply  analyzing  the 
effectiveness  of  a  weapon. 

(7)  There  are  cost-effectiveness  studies  for  evaluating  cost 
with  respect  to  effectiveness. 

(8)  There  are  research  program  studies  for  identifying  areas 
of  high  technical  potential  in  which  R&D  funds  should  be 
concentrated. 

Another  form  of  classification  is  by  model  structure.   Presently,  models 
are  categorized  by  end  use,  i.e.,  truck  models,  tank  models,  artillery 
models,  air  defense  models,  etc.   It  might  be  desirable  to  classify  the 
sub-elements  of  these  models,  since  many  of  the  sub-elements  in  end  use 
models  have  similar  characteristics.   Submodels  might  include  terrain, 
communication,  mobility,  target  acquisition,  etc.   The  classification  of 
these  submodels  and  their  mapping  into  end  use  models  should  be  undertaken 
by  the  AMC  so  that  areas  of  strength  and  weakness  can  be  found  in  the  over- 
all system  of  models. 

We  feel  that  resources  should  be  provided  to  structure  the  technical 
aspects  of  the  model  building  process  to  take  advantage  of  developments  in 
technology  and  concepts.   The  organization  needs  to  allocate  resources  to 
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to  the  problem  of  maintaining  a  central  communication  facility  that  will  al- 
low technical  analysts  to  draw  on  the  work  that  has  gone  on  before,  and  also 
to  provide  them  with  some  perspective  or  overall  policy  direction  in  regard 
to  model  structure.   In  its  simplest  form,  this  might  be  a  decision  to  use 
a  common  computer  throughout  all  of  the  study  processes,  or  at  least  to  use 
a  standard  higher  level  language.   There  could  be  some  guidelines  developed 
along  the  lines  of  the  following  methodology  section  in  this  paper  and  there 
could  be  some  central  software  development  to  support  the  decision-maker 
interaction  that  is  required.   The  resources  might  be  put  first  to  the 
problem  of  providing  the  decision  maker  with  a  flexible  interface  to  the 
model,  or  they  might  be  spent  on  the  problem  of  providing  analysts  with  a 
flexible  interface  to  the  model.   In  either  event,  such  effort  is  likely  to 
be  highly  rewarding  in  terms  of  improvement  to  the  process.   Provision  of 
such  support  for  the  analyst  is  likely  to  result  in  a  very  considerable 
drop  in  the  elapsed  time  for  the  model  creation  activity. 

Conclusions 

Each  of  these  five  recommendations  has  at  its  heart  the  attempt  to 
improve  the  interface  problems  inherent  in  the  four  groups  identified  in 
Figure  1  above.   One  way  of  improving  the  problem,  of  course,  is  merely  to 
recognize  it  and  to  begin  to  work  on  possible  solutions.   However,  it  is 
also  true  that  technology  has  improved  enough  in  recent  years  that  interactive 
display  terminals  tied  to  some  central  computing fecility  over  regular 
telephone  lines  is  a  technically  feasible  and  economically  desirable  pos- 
sibility.  To  exploit  such  techniques  requires  some  effort  on  the  part  of 
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the  organization  concerned.   It  is  not  possible  to  take  previous  research 
projects  or  software  packages  and  apply  them  as  is  to  the  problems  that  the 
cost-effectiveness  studies  in  the  AMC  environment  have.   In  that  sense 
these  techniques  are  only  a  mechanism  to  force  a  rigorous  consistent  evalua- 
tion of  the  role  of  cost-effectiveness  in  the  decision  process  in  AMC. 
However,  our  industrial  experience  indicates  that  this  sort  of  effort  is 
well  worthwhile,  and  the  technology,  although  only  a  means  to  an  end, 
provides  some  very  tangible  and  immediate  benefits  to  all  concerned.   Formal 
models  have  considerable  power  and  potential,  but  only  in  the  hands  of  the 
decision  makers  who  are  eventually  charged  with  the  responsibility  of 
making  the  final  decisions.   Accomplishing  this  task  calls  for  a  consider- 
able degree  of  effort  and  creativity  on  the  part  of  the  organization 
involved,  and  it  is  clear  from  our  study  of  the  cases  made  available  to  us 
that  there  is  a  great  deal  of  potential  yet  to  be  realized  in  the  present 
cost-effectiveness  study  process. 


VII.   Findings  and  Recommendations 
(B)  Model  Methodology 

Introduction 

The  AMC  is  responsible  for  the  selection,  development,  and  procurement 
of  materiel  that  will  best  satisfy  the  projected  requirements  of  the  Army, 
within  specified  resource  constraints.   In  addition,  AMC  is  expected  to 
provide  information  inputs  to  higher  level  agencies  (i.e.  Department  of  the 
Army  and  the  Department  of  Defense)  for  making  decisions  on  alternative 
weapons  systems,  and  on  the  mix  of  systems  appropriate  for  implementing 
defense  strategies. 

Both  these  roles  involve  extensive  use  of  a  group  of  formal  techniques 
variously  labeled  "systems  analysis",  "cost-effectiveness  analysis",  etc. 
We  shall  use  the  term  systems  analysis  (SA)  to  refer  to  these  formal  models. 
The  need  to  use  SA  mainly  stems  from  four  sources. 

(a)  The  limited  capacity  of  human  beings  as  information  proces- 
sors or  decision  makers.   Evidence  suggests  that  the  un- 
aided human  cannot  effectively  cope  with  more  than  a  few^ 
perhaps  five  or  ten  variables,  in  any  decision  situation. 
Systems  analysis  provides  a  framework  for  processing, 
evaluating  and  condensing  available  data  so  that  the 
decision  maker  can  use  the  output  of  such  models  as  an 
input  to  his  decisions. 

(b)  The  complexity  and  rapid  rate  of  change  of  technology  demands 
that  many  variables  (often  hundreds)  be  taken  into  account 

in  any  materiel  decision,  effectively  precluding  the  use  of 
"pure"  judgement  or  intuition  in  such  situations. 
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(c)  The  need  to  allocate  limited  resources  to  the  development 
and  procurement  of  materiel  implies  the  need  to  develop 
measures  of  costs  and  effectiveness  of  alternative  systems 
in  the  context  of  Army  objectives,  potential  or  actual 
enemy  capabilities,  and  the  current  or  future  state  of 
technology.   Such  measures  clearly  require  formal  SA 
techniques . 

(d)  Finally  the  need  for  some  type  of  mental  discipline  to  state 
explicitly  all  the  critical  cause  and  effect  relationships 
and  also  methodically  search  for  and  evaluate  alternatives. 

As  a  result  of  the  above  needs,  SA  has  evolved  to  produce  measures  of 
the  costs  and  effectiveness  of  weapons  systems,  usually  in  the  context  of 
providing  guidance  to  decision  makers  required  to  choose  among  alternative 
systems  for  design,  development,  or  procurement. 

Although  the  aspects  of  cost  and  effectiveness  are  not  independent,  yet 
our  examination  will  consider  only  the  problem  of  developing  effectiveness 
measures  for  models. 

As  a  broad  generalization,  SA  studies  can  be  described  as  "transformations" 
that  accept  as  a  basic  input  the  physical  characteristics  (blueprint  specifi- 
cations) of  a  weapons  system  and  produce  as  output  a  measure  of  the  system's 
effectiveness.   To  support  this  transformation,  additional  inputs  describing 
the  environment  in  which  the  system  is  to  be  utilized,  the  capabilities  and 
strategies  of  both  friendly  and  enemy  forces,  and  the  subjective  valuation 
of  battle  outcomes,  are  required.   Such  a  transformation  process,  in  the 
present  methodology,  is  usually  monolithic.   This  means  that  a  single  study, 
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often  created  ad  hoc  to  deal  with  a  single  weapons  sytem,  contains  the  entire 
transf orr.ia tion  and  attempts  to  provide  a  single  measure  of  effectiveness. 
The  parts  of  these  studies  are  so  interdependent  (because  of  the  monolithic 
structure)  that  the  transferability  of  results  is  limited  extensively,  and 
very  little  of  a  study  can  be  salvaged  if  for  some  reason  changes  to  the 
underlying  assumptions  must  be  made.   Unweildy  structures  also  impose  a 
barrier  between  the  user  and  the  model,  because  the  decision  maker  cannot 
effectively  interact  with  and  benefit  from  the  study.   As  we  have  already 
pointed  out,  unless  the  decision  maker  feels  comfortable  with  a  model^  he 
cannot  successfully  integrate  it  into  his  planning  activities.   He  may  use 
the  models  either  because  of  faith  or  because  of  hesitancy  to  admit  ignorance. 
This  is,  of  course,  somewhat  an  exaggeration  of  the  facts,  but  we  shall 
proceed  to  examine  problems  that  arise  from  the  degree  to  which  this 
monolithic  type  of  study  effort  is  prevalent. 

Findings 

The  shortcomings  of  the  methodology  described  above  are  reflected  in  sev- 
eral classes  of  symptoms  identified  in  the  course  of  our  study.   They  are 
also  common  in  most  applications  of  formal  models  in  industry  as  well  as 
government. 

Changes  in  the  assumptions,  environmental  factors,  objectives,  and  other 
data  inputs  to  a  SA  study,  frequently  invalidate  the  study  conclusions  and 
require  that  an  extensive  new  analysis  be  undertaken.   This  we  found  to  be 
a  common  occurrence.   Such  changes  are  commonplace  results  of  rapidly 
changing  technology,  reformulation  of  strategic  and  tactical  doctrine. 
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shifting  assessment  of  enemy  capabilities  and  intentions,  and,  to  a  limited 
extent,  advances  in  SA  techniques  or  results  of  newer  related  studies.   In 
short,  these  changes  are  a  result  of  the  same  underlying  circumstances  that 
led  to  the  use  of  SA.   Before  the  advent  of  SA  studies  the  decision  maker 
could  not  cope  with  the  many  variables  and  the  changes  which  were  occurring 
in  such  variables.   We  envision  the  day,  if  we  are  not  already  there,  when 
the  user  of  studies  will  be  unable  to  cope  with  the  vast  inventory  of 
specialized  studies.   If  we  are  to  save  time  and  financial  resources  we 
must  find  common  elements  in  SA  studies  and  impose  some  type  of  standardiza- 
tion scheme  on  the  basis  of  these  common  elements. 

A  second  class  of  symptoms  that  we  observed  and  which  relate  to  the  ad  hoc 
monolithic  character  of  SA  studies  consists  of  excessive  study  complexity  and 
the  long  lead  times  required  to  complete  the  studies.   The  former  precludes 
effective  decision-maker  interaction  with  the  study,  since  he  cannot  expend 
the  time  necessary  to  adequately  explore  study  assumptions,  techniques  and 
results.   The  latter  discourages  the  utilization  of  SA  to  make  decisions,  due 
to  the  excessive  time  and  expense  involved.   Both  complexity  and  long  lead 
times  reduce  the  ability  to  improve  on  previous  results,  since  the  results 
are  not  easily  understood  and  may  be  out  of  date  in  any  case. 

Third,  important  value  judgements  are  being  implicitly  made  by  analysts 
in  order  to  complete  the  transformation  process.   Such  value  judgements 
should  be  left,  where  possible,  to  the  decision  maker  who  is  presumably  more 
capable  of  applying  the  required  broader  context  (including  relevant  non- 
quantifiable  data),  than  the  analyst. 
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These  symptoms  can  be  related  to  the  broader  problem  of  providing  ef- 
fective feedback  in  the  decision-maker-analyst  interface,  in  order  to 
promote  learning  from  experience.   Such  learning  is  a  key  element  in 
improving  the  value  and  utilization  of  SA  effort.   In  order  to  provide 
feedback,  a  proposed  methodology  should  help  provide  the  following  specific 
capabilities: 

(a)  Creation  of  motivation  to  act  on  differences  between  desired 
and  actual  study  effectiveness,  by  allowing  identification 
of  responsibility  for  errors  and  areas  where  model  improve- 
ment is  necessary. 

(b)  Reduction  of  lead  times  for  the  study  process  to  provide 
timely  feedback  before  large  commitments  of  resources  are 
made. 

(c)  Means  which  will  enable  the  decision  maker  and  the  analyst 
to  more  fully  explore  the  characteristics  and  sensitivity 
of  model  results  to  particular  assumptions  or  data  values. 

(d)  Greater  consistency  in  data  inputs,  models  and  assumptions 
among  different  studies,  to  help  pinpoint  areas  for  improvement. 

(e)  Capability  to  salvage  parts  of  a  study  in  order  to  save  time 
and  resources,  and  at  the  same  time  encourage  transferability 
of  useful  results. 
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Reconimendations 

In  order  to  achieve  the  above  goals,  we  propose  that  some  AMC  effort  be 
directed  toward  evolving  the  modular,  "building-block"  approach  as  described 
here. 

To  structure  the  SA  process  and  achieve  the  goals  described  above,  we 
first  break  down  the  transformation  process  for  a  single  weapons  system 
into  four  stages,  identifying  the  inputs  and  outputs  at  each  stage.   This 
structure  is  shown  in  Figure  3. 

There  are  several  important  features  of  such  a  structure.   First,  we  are 
implying  parameterization  of  the  input  and  output  variables  at  each  stage  in 
the  transformation  process  for  a  class  of  systems  meeting  similar  program 
capability  objectives,  or  having  similar  physical  or  performance  characteris- 
tics.  If,  in  fact,  the  weapon   systems  were  similar,  the  number  of  distinct 
models  required  to  generate  effectiveness  measures  could,  ideally,  be 
reduced  to  the  four  stages  in  the  process.   Because  weapon   systems  are 
dissimilar,  we  anticipate  a  situation  like  that  shown  in  Figure  4. 

For  a  given  class  of  systems,  several  distinct  models  would  be  required 
at  the  first  stage  due  to  the  lack  of  comparability  (equivalent  parameter 
sets)  for  different  systems.   As  the  higher  stages  in  the  process  are 
reached,  the  number  of  distinct  models  needed  should  be  reduced,  as  each 
lower-stage  model  has  as  its  output  a  more  clearly  standardized  set  of 
parameters. 
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Figure  3:   Multi-Stage  Transformation  Process 
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In  order  to  provide  a  concrete  illustration  of  this  structure,  we  have 
examined  the  mortar  study  ("Effectiveness  Comparison  of  M329A1  vs.  XM571  (RAC) 
Mortar  Shells",  BRL) .   The  parameters  and  variables  involved  at  each  stage 
in  the  transformation  process  of  the  mortar  study  are  shown  in  Figure  5. 

At  this  point,  we  shall  attempt  to  clarify  the  terms  we  have  been  using 
to  describe  the  model  structure.   We  first  distinguish  between  external 
p.Trameters,  which  describe  the  tactical  and  physical  environment  within  which 
the  weapon  system  is  to  operate;  and  system  characteristics,  which  describe 
the  attributes  of  the  weapon  system  itself. 

Within  the  category  of  system  characteristics,  we  have  physical  variables, 
performance  variables,  functional  capabilities,  and  program  capability. 
Physical  variables  represent  the  design  of  the  system;  they  are  the  basic 
characteristics  that  are  expressed  in  the  blueprint  specifications.   Per- 
formance variables  describe  what  the  weapon  can  do  by  itself,  in  a  neutral, 
non-hostile  environment.   These  would  be  similar  to  field  testing  data. 
Functional  capabilities  describe  the  weapon   system  as  it  would  perform  in 
a  battle  environment;  terrain,  target  and  supporting  systems  must  therefore 
be  considered.   Program  capability  describes  the  results  of  a  mix  of  weapons 
systems  and  strategies;  it  thus  places  the  individual  weapon  in  a  broader 
strategic  context,  incorporating  enemy  capabilities,  and  our  own  specific 
military  objectives  with  the  level  of  effort  (#  of  systems)  as  a  variable. 

As  can  be  seen  from  an  examination  of  the  parameters  and  variables  in- 
volved, in  Figure  5  a  very  nearly  equivalent  set  of  models  could  be  used  to 
evaluate  the  effectiveness  of  any  ground-based  weapon   system  firing 
projectile(s)  at  enemy  targets.   At  worst,  we  would  anticipate  changes  in 
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Figure  5:   Mortar  Study  Model  Structure 
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Figure  5  (continued) 
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model  1  to  handle  somewhat  different  specifications,  but  the  set  of  perfor- 
TTiance  variables  required  as  input  to  model  2  should  be  quite  similar. 

We  must  now  examine  the  usefulness  of  such  a  structure  in  the  light  of 
the  required  capabilities  outlined  under  "Findings".   First,  such  a 
structure  will  enable  closer  focus  on  problem  areas  in  the  methodology, 
because  a  model  will  be  used  repetitively.   This  should  result  in  the 
detection  of  errors  and  enable  incremental  improvement  to  be  made  for  each 
model.   Second,  lead  times  should  be  greatly  reduced  because  studies  could 
use  existing  models  "off  the  shelf".   Third,  the  extensive  parameterization 
will  allow  the  testing  of  the  sensitivity  characteristics  of  each  model  with 
respect  to  two  types  of  input  variables,  input  from  lower-stage  models,  and 
parameters  describing  assumptions  about  the  external  environment  within 
which  systems  will  function.   Fourth,  greater  consistency  is  achieved 
directly  through  repetitive  use  of  the  same  models,  forcing  the  analyst  to 
carefully  define  and  classify  data  for  the  relevant  set  of  variables  and 
parameters.   Finally,  a  data  base  for  models  can  be  built  which  will  allow 
fast  access  to  tlie  existing  inventory.   The  latter  will  be  classified  on 
the  basis  of  the  various  model  stages  and  by  relevant  characteristics 
(external-environmental  and  internal-system  characteristics).   Such  a 
classification  scheme  will  be  very  useful  when  the  link  between  models  and 
on-going  information  systems  is  established.   For  then,  we  will  be  able  to 
integrate  parts  of  studies  and  also  develop  schemes  for  updating  the  data 
base  automatically  with  each  new  entry. 


>l 
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Implementatlon  Considerations 

We  now  consider  the  steps  necessary  to  implement  the  re-structuring  of 
SA  efforts  described  above.   First,  existing  SA  studies  should  be  examined 
in  the  same  way  as  the  mortar  study  example,  to  determine  the  required  sets 
of  standardized  variables  and  parameters.   A  sufficiently  broad  range  of 
studies  is  required  to  insure  that  the  resulting  specification  of  model 
structure  is  relatively  complete.   Also  it  will  be  necessary  to  go  through 
an  extensive  examination  of  existing  studies  to  justify  the  choice  of 
parameter  sets.   The  range  of  studies  should  include  systems  with  functional 
capability  in  the  areas  of  sustenance,  transportation,  communication,  target 
engagement,  and  intelligence.   Due  to  the  lack  of  consistency  in  presenta- 
tion of  study  results,  to  which  we  addressed  ourselves  before,  the  amount 
of  time  required  to  isolate  the  relevant  variables  and  parameters  is  likely 
to  be  large.   Second,  a  system  for  standardization  of  input  data  (both 
external-environmental  and  internal-system)  must  be  developed  and  standardized 
formats  for  the  collection,  use,  storage  and  update  be  specified.   These 
formats  should  take  into  account  the  classification  of  variable  and  parameter 
sets  evolved  above,  as  well  as  the  form  and  availability  of  the  data. 

Third,  the  data  base  to  be  utilized  for  model  input  should  be  organized 
along  the  modular,  hierarchical  lines  proposed  earlier.   This  will  facil- 
itate updating  of  the  data  in  response  to  changes  in  the  environment,  and 
re-structuring  of  such  to  fit  new  models  as  they  evolve. 

Finally,  a  program  to  provide  continuous  evaluation  and  feedback  of  model 
validity  and  utility  should  be  established.   As  other  sections  of  this  report 
suggest,  such  a  program  will  require  meaningful  participation  on  the  part  of 
both  the  decision  maker  and  the  analyst. 


VIII.   Possible  Paths  for  Future  Research 

The  activities  that  we  carried  out  last  year  and  our  exposure  to  various 
agencies,  people  and  studies,  suggest  a  number  of  useful  research  possibil- 
ities for  the  future.   These  can  be  explored  no  doubt, by  agencies  within  the 
Army.   However,  the  day-to-day  pressures  from  the  operating  environment  and 
the  history  of  institutional  settings  serve  as  an  impediment  to  the  success- 
ful pursuit  of  some  of  these  activities  that  we  identify  here,  if  these  are 
carried  out  internally.   We  believe  that  a  joint  effort  by  the  Army  and  our 
research  team  will  be  very  productive,  and,  for  this  reason,  we  are  ready  to 
undertake  a  continuation  of  this  research  for  another  year  and  concentrate 
on  some  of  the  topics  listed  below. 

The  order  in  which  these  projects  are  listed  reflects  our  notions  of 
priorities.   Bearing  heavily  on  our  judgement  were  such  practical  issues 
as  the  utility  to  the  Army  and  the  feasibility  of  researching  the  topic  at 
some  satisfactory  depth  within  a  year.   We  do  not,  of  course,  propose  to 
attack  all  these  topics  during  the  next  twelve  months.   The  amount  of  ef- 
fort required  for  such  a  task  far  exceeds  our  available  resources.   Only 
parts  A  and  B  might  therefore  be  contemplated  for  next  year. 

A.   A  Conference  to  Interface  Decision  Makers  and  Systems  Analysts 
In  the  course  of  conducting  our  research,  we  were  impressed  by  the  gener- 
al availability  of  talent  within  the  organizations  which  perform  studies  for 
the  Army.   It  is  also  clear,  however,  that  the  work  setting  is  not  conducive 
to  meaningful  exchange  and  cross  fertilization  of  ideas  among  representatives 
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of  different  groups  involved  in  cost-effectiveness  analyses.   The  Review  Board 
setting  is  not  intended  to  provide  the  medium  for  improving  study  methodology 
and  analyzing  the  problems  facing  decision  makers  and  systems  analysts. 

For  the  above  reasons,  we  believe  that  it  would  be  fruitful  to  invite  to 
MIT,  the  decision  makers,  the  directors  of  major  analysis  organizations  and 
the  proponents  of  various  methdological  approaches  and  exchange   ideas  for 
the  improvement  of  studies  and  of  the  overall  study  process. 

In  order  that  such  a  conference  be  effective,  position  papers  on  critical 
problems  must  be  prepared  and  circulated  ahead  of  time.   The  conference  itself 
will  be  devoted  to  discussion  on  the  papers  and  to  the  summarization  of  find- 
ings.  The  papers,  discussion,  comments  and  integrative  remarks  could  then 
be  put  together  for  wider  distribution. 

B.   Develop  Methods  for  Building  Flexible  Models 

Our  research,  so  far,  convinced  us  that  mangerial  decisions  within  AMC 
and  the  Army  are  very  sensitive  to  quantitative  models.   The  opposite,  how- 
ever, is  not  always  true.   Because  of  weak  interfaces  between  the  decision 
makers  and  (a)  the  systems  analysts,  (b)  the  models,  and  (c)  the  data 
bases,  we  find  that  there  is  an  urgent  need  to  improve  the  sensitivity  of 
the  models  to  the  needs  of  the  decision  maker.   One  part  of  such  effort 
involves  improvements  in  model  methodology. 

In  another  part  of  this  report  we  suggested  a  "building-block"  concept 
for  structuring  cost-effectiveness  studies.   The  purpose  of  that  suggestion 
was  to  create  flexibility,  reduce  the  time  required  for  completion  of  a 
study,  and  encourage  updating  and  transferability  of  the  results  of  studies. 
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We  are  happy  to  note  that  ATARS  is  using  one  such  approach.   Furthermore,  the 
techniques  developed  for  input-output  analysis  (matrix  approach)  offer  some 
promise.   No  doubt  there  are  other  techniques  which  use  the  principles  of 
transformation,  decomposition  (modularity),  and  hierarchical  structures 
which  merit  attention.   For  these  reasons  we  believe  that  further  research 
is  required  for  the  development  of  the  necessary  theory  and  methodology  for 
guiding  the  activities  of  AMC  study  agencies. 

We  propose  to  work  closely  with  the  agencies  performing  cost-effectiveness 
analyses  in  the  development  of  the  "building  block"  and  "matrix"  approaches 
discussed  in  our  reports.   In  developing  the  former  concept,  we  will  attempt 
to  identify  in  conjunction  with  the  AMC  personnel,  the  most  logical  and 
suitable  points  for  placing  standard  transformations  and  hierarchical  breaks 
in  the  AMC-wide  model  structure.   We  will  also  attempt  to  develop  the  neces- 
sary framework  for  classification  of  inputs  and  outputs  necessary  for  the 
successful  use  of  such  models.   A  logical  extension  of  the  "building  block" 
approach  will  be  an  application  of  some  of  the  conepts  of  input-output 
analysis  to  identify  the  relationships  between  Army  objectives  and  various 
weapon  systems,  and  between  the  weapon  system  submodels  and  the  complete 
weapon  systems. 

The  results  that  we  envision  emanating  from  such  a  capability,  as  described 
above,  are:   reduction  of  the  length  of  the  study  process,  faster  response, 
reduction  of  the  frequency  and  cost  of  revisions,  learning  through  informa- 
tion transfer,  and  evolution  of  a  classification  framework  which  will  pave 
the  way  for  projects  "C"  and  "D"  as  described  here. 
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C.  Develop  Classification  Schemes  for  Studies  and  Construct  Efficient 
Data  Bases 

Transferability  of  results  is  a  must,  if  learning  from  the  experience  of 

others  is  to  take  place.   To  unlock  the  tremendous  power  which  now  remains 

mostly  within  the  study,  a  common  scheme  for  classification  and  updating 

must  evolve  as  well  as  a  procedure  for  maintaining  a  common  data  base.   A 

research  group  can  work  together  with  decision  makers  and  systems  analysts 

to  develop  such  schemes  for  classif icati rn.   From  then  on,  each  study  shall, 

among  other  things,  include  a  page  with  information  pertaining  to  the 

adopted  classification  scheme  and  across  the  various  dimensions  which  were 

chosen  for  this  purpose.   This  information  will  be  used  both  for  cataloguing 

the  study  itself  and  also  updating  the  common  data  base. 

D.  Designing  an  Information  System  for  Controlling  the  Study  Process, 
and  Linking  the  Studies  with  Operations 

In  our  findings  we  have  pointed  out  some  of  the  advantages  of  formal 
planning  and  of  linking  formal  plans  and  operations  through  managerial  in- 
formation and  control  systems.   A  very  useful  project  for  the  Army  Materiel 
Command  will  be  to  undertake  the  development  of  such  an  information  and 
control  system.   Its  objectives  should  be  to  develop  and  implement  procedures 
by  means  of  which  replanning  will  become  an  integral  part  of  the  continuous 
control  process.   The  replanning  and  control  that  we  have  in  mind,  involves 
both  that  which  is  a  part  of  the  study  process  and  the  activities  which  are 
initiated  as  a  result  of  the  implementation  of  the  recommendation  of  studies. 

The  development  of  such  an  information  and  control  system  (to  obtain 
meaningful  feedback  from  the  point  of  initiation  of  a  study  to  the  point  of 
expiration  of  its  final  consequence)  will  involve  extensive  investment  over 
a  period  of  at  least  three  years. 
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E.  Simulation  of  Different  Approaches  to  Cost-Effectiveness  Analysis 
Decision  makers  today  must  have  at  their  disposal  systems  which  allow  in- 
teraction.  In  this  way,  alternative  plans  may  be  tested  and  knowledge  gained 
on  the  basis  of  hypothetical  cases.   Visual  display  units  with  a  capability 
for  graphical  presentation  of  results  are  available  and  in  current  use  so 

the  Army  must  now  start  planning  for  the  future.   These  devices  provide  an 
effective  interface  between  the  human  decision  makers,  the  models,  computer 
technology  and  data  banks. 

For  such  an  approach  to  systems  analysis,  the  efforts  of  the  decision 
maker,  the  study  preparation  agency  and  MIT  must  be  brought  together  and 
applied  to  one  or  two  studies.   If  successful,  these  experiments  will 
enable: 

1.  The  decision  maker  to  develop  more  effective  plans 

2.  Researchers  of  the  management  to  gain  insights  into 
the  planning  process  and  move  toward  the  develop- 
ment of  intelligent  information  systems  and 

3.  Systems  analysts  to  develop  standard  approaches  for 
different  kinds  of  studies. 

F.  Tests  for  Sensitivity  of  Decisions  to  Model  Complexity 

The  present  system  generally  leaves  it  up  to  the  analyst  to  decide  how 
complex  the  study  should  be.   A  research  program  could  be  undertaken  to 
evaluate  the  effect  of  complexity  and  model  sophistication  on  the  quality 
of  decisions  produced  by  the  decision  maker.   One  way  to  carry  out  this 
project  is  to  conduct  two  parallel  programs  of  different  complexity  focused 
on  a  particular  problem  or  decision.   The  results  of  these  two  programs, 
prepared  independently,  could  then  be  evaluated  to  determine  what  benefits 
sophistication  offers  and  what  criteria  might  be  used  for  deciding  the 
level  of  complexity  of  a  given  study. 


Attachments 


Attachment  1 
Study  Hypotheses 

1.  Decision  makers  expect  that  the  models  will  make  decisions  for 
them  rather  than  helping  them  to  make  the  decision. 

a.  They  do  not  participate  in  the  directive 
writing  process  or  in  the  face-to-face 
negotiation  of  project  objectives  and 
methodology. 

b.  They  do  not  participate  in  proposal  review. 

c.  They  donot  participate  in  analysis  review. 

2.  Variables  outside  the  scope  of  the  cost  effectiveness  study 
have  a  significant  impact  on  a  decision.   That  is,  in  many 
cases,  events  which  were  not  examined  in  a  cost-effectiveness 
process  are  found  to  be  governing. 

3.  The  study  time  before  a  military  systems  decision  is  made  is 
longer  than  necessary. 

a.  Most  studiesare  re-done  to  examine  conditions  not 
previously  specified  by  the  decision  maker,  either 
because  the  decision  maker  did  not  take  the  direc- 
tive setting  process  as  a  significant  part  of  the 
planning  activity,  or  because  there  were  changes  in 
the  environment  which  were  not  possible  to  forecast. 

b.  Most  studies  which  are  re-done,  are  re-done  to  examine 
new  conditions  that  were  not  previously  specified. 
That  is,  there  could  have  been  higher  level  inputs 
from  senior  decision  makers  which  were  known  if  the 
decision  makers  involved  had  been  asked  to  parti- 
cipate in  the  directive  setting  process. 
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4.  There  is  no  development  program  to  increase  the  study  capability. 

a.  Ad  hoc  studies  use  most  of  the  available  study 
resources,  which  leaves  little  or  no  time  for 
basic  research  in  study  methodology. 

b.  Studies  tend  to  be  treated  as  unique  rather  than 
providing  an  opportunity  for  making  progress 
toward  a  greater  integrated  capability,  or  in- 
creasing the  understanding  of  those  involved  in 
that  class  of  model.   This  is  the  case  because  the 
resources  applicable  to  this  generalization  and 
study  methodology  process  are  not  adequate. 

c.  There  is  no  formal  information  feedback  system  for 
continued  improvement  of  study  quality. 

5.  The  assumptions  of  the  study  often  turned  out  to  be  inadequate 
thus  affecting  the  final  decision. 

6.  These  studies  are  not  flexible. 

a.  The  models  are  not  built  to  include  a  decision- 
maker model  interface. 

b.  Lack  of  a  systems  or  hardware  interface  results  in 
a  data  bank  that  contains  data  recorded  on  the 
wrong  media  or  in  the  wrong  format.   This  results 
in  a  lot  of  time  and  effort  being  spent  in 
transforming  the  data  or  in  a  new  data  generation 
program. 

c.  Models  are  generally  not  parameterized. 
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Attachment  2 


ProjKCt  ytanafi^^v   or  S  i.udy  Mnna;;tig  Qtieiiiiionnalre 


TiCl*^.   of  Study 


Da  tit 


Source  of  Data  or 
Person  Interviewed 


Position 


■(„     yiiat  vae    th-w   inicially  »2<-ttujat<r.d  cofitV 


2       What  araouni   of   funding  was   apprcrv?tc1? 


3       V7ho  Jiwdi.    Sht    fundi ng  dsci/^iion'i' 


■^       What  criceric  w«rt-  uycd   la  ^VrTliiat iny   the  appTopr^«::e    i«vt' 1   uf   funding 
i  or    tht    iludy?      (Agr-ncy  »j;UC'!e,    <i£udy   priority,,  magnjtud^   of   dtci^ion 
Co  be  made^,   aviiilabxlily  o(   nt'iffj    bco^!-^  of   cask,   ate,) 
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5,      WnJ.ch  major  crganizationai    elpvc.tntj  within  AiC   provided   (-^arsonnel,    tlmn:. 
or   mtomsation   xu  the   study? 


6       Which  Army  CoCTm.indA   or   other  i.ivni  iaxTy   iargy   jrpanitaLlonal   eiemant* 
within   the  Army    tuch  as  DA  StaLf   prc/vi.de<3   ptTiOnne!  i,    timt  or   in^ 
fortuation   to  the   »tudyy 


How  many   othk^r  military  serviced   ,»Jc»vi'J«:ti   feraonntel,    tlmt    or    information 
to   the   3Cudy? 

""^^  Office  of  .iec>    of  I»ef. 

Air  Forct= 


CIA 

ftarlnefc 

Coa.-ji    Guard 

National   Guard 

Othtr 

8       '..'!i<u  waa   the    IvPc'crship  appxoach? 

--__..   ^'^'     d?ri.-c;or  x^ith   compitic    authority 

H    learn   LOcrdini\i.ot 

fidfll   cr    )oxnl.    icadersiDjj 

OS  h^-.r 
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9   What  wert  the  rharncr.erif  tlco  of  th«j  team  leader 

Civilian     Military 

Rank 


Educatton  DKgret-'-. 

Fields 


Year  recfcivcd  bt«£hGlor';i  degvet' 
Ext'crJenct 


12       How  many  alternative   yysttma  v,ere  con«Adfcrt?d 


l3.      Why  wert'   thefts   aiteiriativt'iJ   s«-ifCt<idV 


»4       Which  pv-i-foitnance  criteria  weca  able   Lo  be  converted   into   financial 
meesiirt.  :i.' 


Which  Wi=rii   th*r  mciatutes  which  could   not    be  convtrcfcd    Into   financial 

tneaoureiSi' 
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15        How  many  paramt tt;rs  werti  examiined?      Liat   if   pojsBible 


T6        F-aw  many  environmt-ntat   conriition:!  wert  Kxaroined?     List   if   poaaibld 


17       Wa.j   a   dpecific   atteini»t  aiacie   to  rwlatt:   not   conuiituauratfc  criteria? 

by   ranking 

hy   usjn{.>   ^   t'jLgurp    of  mtcit; 

by   ..^iiti'ig  vDlnimuiQ  ai  IcT/v'able    levtl  •   on   i^nch   separate   crvttria 

iTlttiTia  v/v  re    ieli    uniclattu 

othtv 


18       Wfi--?   a   otnsitivlty  analyst  i   m.jdft  of  any  oi.    \.h>.'   parROKctird   in   the   study'' 

lifw  ixiany 

W)  1  ch    ouG;6  i' 
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19  ,     Vlh&t   pjrr  ci?nt   of   the   paramet.irw    m  thtf  tnodei  could  b*   changed   so  that 
new  cost/benefit  rankings  could  be  piovid<-d   in  a  week? 


20       What  data  which  you  did   not  have  would  havii   dignlficantiy   improved   the 
studyf 


2).,      What  i7a>i   the  extent  and   sourcfc   of  data  v^hich  had   bten   previously  col  it -.'ted 
and    recorded;' 


22.      HW'?  imich   t"iro<.   weuld   It    cakt-    Co  change  a  railor  non  co^t  a»3uraption  ot 
the   i-fudy  and   rt   tvaluate   the-    :  one  1  us  ion' 
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73       Ifliat  werti    th«  uio»t  unsupported  asaumpciona   in   th*  anaiyala? 


2A       Whac  w«L.   the   actuni    time  and   coiJt   irftijuired   Zvr  corduct  o£    Lhe   study? 


25        How  acl»i:^uat&  woi   the   time   provided    for  the  conduct   of   tht^  atudy; 

more  a 

ihKn                                                 IttvJfe  vtry 

adequ.Ttfc            ade^juate            inadt(,uate  i.rmdc<iUdi<e! 

(      )                 (      )                        i      )  (      ) 


2(>  .      W«s    it    pob'Jibic    to  an-JW^T  aJJ    thfi  \;u<r.'8f  ion*>    m   the    study  dn-ective 
comp;<et«Iy? 

How  iiyjnyV 
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27.      \^at  wfio    the    conclusion   of    thf    Atudy"! 


28o     Wr3   one    a.:  v.r;rn«ti.ve   recoffimenritd  ov*  r  all    tht   rjthtfrs? 
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peelsiop  Ik-.ke.r   Queatxonna5..ra 
Tl.tl«5  of  Study 


Date 


Sourca   of  Data   or 
Person  lnt<ir\^iewed 


FostCion 


1,      Vtljy  wQji   the  study  intttatfed? 


2.     Vjbat  liUgnltude  of  declBion  wp.9  perceived  al   this  point  as  being  et  stakes? 


3.      Vihat  werti   the  rao"(£  unsupported  aasTinptionti   tii  the  analyai>?? 


<!*.      v-Jhich  q\»ei;i:ionG   in   the   study  diruccivcs   or  vr"vk:tafc<snenL   couid   not  bis 
cotapiettly  RnriT*crfcd? 

Ho";.^  TSr'.ny'i 

5v      How  rr.any   revie;^w  were  tiiadc-i  of   tnc-;  validity  of   fcho   B;:udyo   by  v;hora' 
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To  what  extent  x*ere  the  findings  of  the  study  surprising  or  unanticipated 
in  ccraperleon  to  the  expectations  of  people  before  che  study  was  mads? 


very 
surprl.^ing  *- 


obvious 


7        How   thorough  wa*i   thi«3   study? 


very 
thorough     f— 


not 
-i     thorough 


a       Hotj  good  waJ    the  trie thodo logy? 
appropriate    / '■ 


J     inappropriate 


9       How  good  wsb"   the   interpx'etstion  ot   the  analysis? 


ver-y 
oood  ! — 


_i 


./ 


J      pOOT 


10       How  good  v;as   the  study  in  meeting  the  decision  requlrementfi? 


very 
good     '" 


-H 


p007- 


11        Hew  strong  wea   th^;^   recoiraiendatiLon  of   fcht   study  relfttive   to  the   othar 
cO!i>=i '.derations    ta  luaktng   tht  deciaion? 


very 
dtrong 


very 
--.'    vedk 
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12   What  decision  was  tuade  as  a  reault  of  the  study? 

Buy 


^  Partial  or  Pilot  buy 

_  "ifi^t 

_  Walt 

Restudy 

Reduce  k  program 

Tormina ta  a  program 


_^ Changt  a  policy 

Other 


Stai:e  in  gre^atet  detail  and  why  was    the  dec  is  ion  taade. 


Ill       Whst  additional   infoniiKtlon  had   to  be   obtained  and  conaidered   in  order 
to  make   the  decj.vionV 


I'i       Were   thert;  politi::al   con.'. ide^at ions? 
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15       What  was   Che   shor£  run  end   long  run   funding  magnitude  of   the  dfecldion 

made? 


16.  Who  was  Che  deci.^ion  maker  and  what  wag  his  rank  a^id  posltxonV 


■J 7   Itow  high  did  the  study  get  briefed? 


18.   To  what  extent  did  the  decision  ins.de   ngree  with  th«  conclusi.on.'i  of  tht 
d  tudy ? 


19   I'That  auxiliary  .:*Cudlea  and  decisiona  wers  required  as  a  r-esult  of  the 
^iLudyi' 


identify  the  purpose  of  each^ 
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20.  What  were  the  recooaoendationH  of  the   later  studies? 


21   To  what  extenf;  did  the  dacifiion   tuiiktr  participate  :.n  any  part  of  the 
execution  o£  the  study V 


22   What  actlona  were  taken  aubaequent  to  tht  initial  decision?  Give 
dates  for  t;ach  action. 
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23       To  vhat  exeent  wsre   the  asi^unipclonti  and  conclusions  of  the  study  borae 
out   in  later  avsnta? 


2A       VJas   the  decision  later  revsrsedV 
yea  ^.         no 
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Queattonnatre  for  Cogntaant  Office 


Title   of  Study 


is.>a2e 


Source  of  Date  or 
Per. •son  In tervit.we<1 


Position 


1        Wlio  i.i-»   the  Anny   initiatfesd   the  ret^nemi:  fur   th«   .-ctudy' 


,VjC . 

SygUftOT  l»i-*>j&ct  Office- 

Another  Cocmiand 

Another  3ervic8 


DeparfTiifint  of   chti  Array  Starr 
DfcpnrtJfient  of  Deftnae 

Other 


2:      VJhat:  ptopli'  wert^  in  the  chain  of   the  .strudy  directive   (/-"JorksCataitwsnt)    frow 
initintor   to  thii  ai:udy  p}jrrormt:rV 


3.     What  vjfis   thi^  naJrure  cf  the  direit-Civ:'  to  perform  the  study? 


i^,.      I'lCTW  wany  c-vieflfeiono  were  asltec'? 
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5       ilot-?  many  a&fvnnptions  oi"  condition*  v)<4rtr  gtV'in? 


6       Wt-rt   the  alternative^  aysfctms  which  wert;   co  be  contsiderad  in  the  atudy 
ipeciflpfl  in  ihfc  directive? 


7r      Hot;  much  was    the  directive  modified  bti'forfe   fch>4   »itudy  wa6  undertaken? 


8^     What  were  the  sources  and  reafconfi  for  t-.h«  change? 
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9       Wliac  alternative  organization^  were   consl.deriid   for   the  etudyJ' 

Ynduatry 

Not   Cor   orofit 


University 

"Oep/irt;ment  of   the  Arasy  Staff 

Departrct'-ut   oc    f:he  'Jei'tsoAa   Task  Foy.-ctf 

A  partjtculftr  Army  Command 


3  0   Which  group  wsr  ohoofn  and  whyV 

Particular  exjiertj&e 

Funding 

Juri?-diction 

Current  workload 

Oi.h'^r 


ii       Who  utad*.   rhi&   aclection? 


i2,.      Did   the   i'.hoeen  «tudy  urg«ni>ra«:lon  have  any  inttresl:   in    the  decision 
to  bfj  tTu-.de  on   the  basis   of   tht   study? 

Yt2s<  No.   Whuu 


What  wai^  their  preferrtid  aolution  oi:  concluaioni' 


U,      U'Jiat  wac   th*.>   initially  «?:utiiaated  co«t? 
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14  .      Whflt  amount:  of   irwnding  was  ap^srorvedV 


).5  -      ^jJho  madfj   the   funding  decisionT 


16       What  cr.*.t;erf.a  were  used  in  evaluating   the  ap|,yjfopriatt  level  of  funding 
Cor   tht  study?      (Agency  qnottv    afcudy  priorrtyu  Magnitude  of  dei;ialon 
to  be  mndsp   availability  ot   staffs    ecopt  of   task,   etc,) 
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i7c     v;hat  was   rha  r.ctual   tinw  f.nd  co3t  rtcjuiricd  for  conduct  of    Che  aLudy? 


18       V.ov  Many  reviews  w«re  madei   of   ihe  validity  ot    thf  study,    by  whofnY 


19       For  each   revit'w  which  of   tht^  foilfjwing  rtbCcsBTitndatJ.ona  wtre   taicenl' 

12  3 

Accept  fche  study 

^^ ReijjTibSt  cliff >srenL  cjuithcioiogy 

^^  Coritct  ctimpuCiifcloaal   trror 

Request    study  oi^   other  conditj.on}« 

_ Rt^juesi:   rteaj  data  bh  con.iij.dered 

Rev:juei<t  paraiiel   etrfly  by  another  agency  be  made 

_^ Rsque^t  nev?  sl'iikx-nafcive'i  be  analymia 

_».„„_  Rt'viMt    interprir taCikJit  of  riisults 

R^jfcCt    »f:udy 


Ofchr;?- 


20       Why  wa'^   "drii,  recijnuiiindation   taken? 
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21       What  decision  was  made  after   che  yfcudy» 

Buy 

Partial  or  Pilot  buy 

Test 

Wai  t 

Restudy 

^_^ R'iduce  a  prograin 

Ttnainatt;   a  profjrniii 

_____  Change  a  policy 


Statti   la  gi.*<»ater  dttail  and  why  was   thXe  fiw«.-.i^.ioa  made. 


22-      WTio  wa«   the  drciaion  maker  and  what  wa^  his  renk  and   i;ot.ition? 
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23c   ^^Bt  auxiliary  studies  :id   decisions  wtr^  required  aa  a  result  of  the 
fltudy? 


Identify  the  purpose  oi'  eachc 


24.,  What  were  ^c  recoramendatlona  of  the  later  studf.eal' 


23o  Na'.ri  each  «itudy  which  preceeded  thes«:  studies. 


26..  What  was  the  outconte  of  each? 
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27   Wiat  actions  veve   taken  subsai^uent  to  uhe  Initiai  decision?  Give  dates 
for  each  action? 


28„   lo  what  extent  were  the  ^>-!!  sumption  a  and  concliiaiunt,  of  the:  study  borne 
out  in  IcTLer  events't 


?.Sr  Was  the  decision  later  reversed? 
, yea no 

30.>     I'Jlion  was  the  first  otiidy  in  this  serico   inifciaced? 
31       Wlien  wf.8   i.he   last  study  completed? 


Attachment  3 


List  of  Studies  with  Suitable  Identification 


1.  Cost/ Performance  Analysis  of  M656,  M520,  M54A2  and  M548  Vehicles, 
August  1967,  U.  S.  Army  Materiel  Command 

2.  A  Cost  and  Effectiveness  Evaluation  of  Sand  Bag  Textiles  (U) 
December  1967,  Memorandum  Report  No,  1882,  BRL,  RDT  &  E  Project  No.  1P523801A098 

3.  M561  Cost  Effectiveness  Study,  Revised  September  1966,  Technical 
Study  EEB-662,  U.  S.  Army  Tank -Automotive  Command 

4.  Parametric  Analysis  of  Pershing  QRA  Alternatives  (U),  March  1967 

5.  Cost-Effectiveness  Analysis  of  a  Non-Nuclear  Sergeant  (U) , 
8  November  1966,  Revised  18  October  1967,  Report  No.  RF-TR-66-2, 

DA  Project  No.  1X279191D687,  U.  S.  Army  Missile  Command,  Redstone  Arsenal, 
Alabama 


6.  An  Effectiveness  Comparison  of  4.2"  Mortar  Ammunition  M329A1  VS. 
XM571  (U),  June  1967,  Technical  Note  No.  1661,  RDT  &  E  Project  No. 
1P523801A098,  BRL 

7.  A  Single  Versus  Twin  Engine  Helicopter  Troop  Carrier  Study  (U) , 

19  January  1968,  Technical  Report  02015.04-1,  Contract  DA-49-186-AMC-393 (X)0, 
Vitro  Laboratories,  Silver  Spring,  Maryland 

8.  Exhibits  Law  Workshop  (U) ,  7  July  1967,  Revised  Summary  Report 
Law  Workshop  (U) ,  22  September  1967  (This  report  supercedes  report 
dated  7  July  1967) 

9.  Reval  Wheels  (Re-evaluation  of  the  Army  Tactical  Vehicle  Program) 
Final  Report,  1  March  1968,  Assistant  Chief  of  Staff  for  Force  Development 

10.   A  Study  of  the  Medium  AntiTank/Assault  Weapon  (U) ,  August  1963, 
Memorandum  Report  No.  1488,  RDT  &  E  Project  No.  1M523801A286,  BRL 
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ll. Technical  Review  of  Army  Air  Defense  System  (TRAADS),  15  March 
1968,  Technical  Review  of  Army  Air  Defense  RDT  &  E  Program  Review  Plan 
28  December  1967   U.  S.  Army  Missile  Command 


12.   Advanced  Forward  Area  Air  Defense  Systems  Study  Phase  1  (U) , 
15  March  1966,  Report  No.  RD/RE-SS-66-1,  U.  S.  Army  Missile  Command 


13.   Parametric  Design/Cost  Effectiveness  Study  for  a  Mechanized 
Infantry  Combat  Vehicle-1970  (MICV-70)  (U) ,  Final  Report  Vol.  1-3, 
Report  No.  CM- 2144-H-lb,  Contract  DA-11-199-AMC-651 (W) ,  November  1966, 
Cornell  Aeronautical  Laboratory.   Also,  Cost  Effectiveness  Analysis  of 
a  Strong  Anti-Armor  Capability  for  MICV-70  (U) ,  Final  Report, 
30  June  1967 


14.  Dispersions  for  Effective  Automatic  Small  Arms  Fire  and  a  Comparison 
of  the  M-14  Rifle  with  a  Weapon  Yielding  Effective  Automatic  Fire  (U) , 
January  1961,  Technical  note  No.  1372,  BRL  (not  complete) 

15.  A  Cost/Effectiveness  Study  of  Mobile  Utility  Module  System  (Mums  18-5), 
March  1968,  U.  S.  Army  Mobility  Equipment  Command  (not  complete) 

16.  An  Operational  and  Cost  Effectiveness  Study  of  the  Lance  Missile 
System,  Combat  Development  Command  and  Army  Materiel  Command,  April  1965, 
Vol.  I-VI 


17.   Study,  Optimum  Mix  of  Artillery  Units  1971-1975,  Combat  Development 
Command,  July  1967 


18.   Re-optimization  of  a  Multiple  Artillery  Rocket  System--Mars  II, 
BRL,  May  1964 


Attachment  4 

Ranking  of  Studies  Based  on  Composite  Index 

Reval  Wheels  Excellent 

Redleg 

Pershing 


Sandbag  Very  Good 

Law 

Mars  II 

Traads 


Lance  Good 

Twin  vs.  Single 

FAAADS 

Grenade  Launcher 


MAW  Adequate 

MICV-70 

M  656 

M561 

Mortar 


.•-w 


"  "  ■    le     •" 


Date  Du( 


ITPTB*^ 

'fSfrrh 

? 

% 

^■"■^  '  G 1382 
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